Systems and methods for providing user tagging of content within a virtual scene

ABSTRACT

Systems and methods for providing user tagging of content within a virtual scene are described. One of the methods includes sending data for display of a virtual environment on a head-mounted display. The virtual environment includes a virtual item. The method further includes receiving an indication of a selection associated with the virtual item. The method includes sending option data for allowing entry of content regarding the virtual item upon receiving the indication of the selection, receiving the content, associating the content with the virtual item, and sending tagged data for displaying a tag associated with the virtual item.

CLAIM OF PRIORITY

This application claims the benefit of and priority, under 35 U.S.C. §119(e), to U.S. Provisional Patent Application No. 62/357,342, filed onJun. 30, 2016, and titled “SYSTEMS AND METHODS FOR PROVIDING USERTAGGING OF CONTENT WITHIN A VIRTUAL SCENE”, which is hereby incorporatedby reference in its entirety.

This application is a continuation-in-part of and claims the benefit ofand priority, under 35 U.S.C. § 120, to U.S. patent application Ser. No.14/206,219, filed on Mar. 12, 2014, and titled “SWITCHING MODE OFOPERATION IN A HEAD MOUNTED DISPLAY”, which is hereby incorporated byreference in its entirety.

The U.S. patent application Ser. No. 14/206,219 claims the benefit ofand priority, under 35 U.S.C. § 119(e), to U.S. Provisional PatentApplication No. 61/832,758, filed on Jun. 7, 2013, and titled “SWITCHINGMODE OF OPERATION IN A HEAD MOUNTED DISPLAY”, which is herebyincorporated by reference in its entirety.

This application is a continuation-in-part of and claims the benefit ofand priority, under 35 U.S.C. § 120, to U.S. patent application Ser. No.14/658,123, filed on Mar. 13, 2015, and titled “METHODS AND SYSTEMSINCLUDING TRACKING HEAD MOUNTED DISPLAY (HMD) AND CALIBRATIONS FOR HMDHEADBAND ADJUSTMENTS,” which is hereby incorporated by reference in itsentirety.

The U.S. patent application Ser. No. 14/658,123 claims the benefit ofand priority, under 35 U.S.C. § 119(e), to U.S. Provisional PatentApplication No. 61/953,730, filed on Mar. 14, 2014, and titled “METHODSAND SYSTEMS TRACKING HEAD MOUNTED DISPLAY (HMD) AND CALIBRTATIONS FORHMD HEADBAND ADJUSTMENTS”, which is hereby incorporated by reference inits entirety.

FIELD

The present disclosure relates to systems and methods for providing usertagging of content within a virtual scene.

BACKGROUND

The video gaming industry is gaining traction. With the advancement oftime, more people are playing video games using gadgets, e.g.,controllers, cell phones, head-mounted displays, etc. To play the videogames, information should be communicated.

It is within this context that embodiments described in the presentdisclosure arise.

SUMMARY

In some embodiments, methods for tagging virtual-reality content inshared spaces that are visited by different head-mounted display (HMD)users are described.

In various embodiments, tagging of content includes leaving recordedcontent or voice memos that are listened to by other HMD users when theyapproach the content or select the tag.

In several embodiments, a tag includes dropping a virtual asset into avirtual-reality scene, and providing a record of the different HMD usersthat interact with that dropped virtual asset.

In some embodiments, virtual tags are created using voice commands. Thevoice commands allow users to request specific types of content to findin specific virtual-reality environments.

In various embodiments, some tags in a virtual reality environment areobtained dynamically by a system, and categorized so that othervirtual-reality users can find that content and add to the tag orcomment regarding the tag. The more content is interacted with in thevirtual environment, the more that content is categorized, identified,and recalled by other HMD users.

In some embodiments, certain content in a virtual-reality environment iselevated to more interesting than other content. Certain content isproposed to new HMD users entering the virtual-reality environment. Ascontent becomes more popular, the content is pushed to other HMD usersas they enter the virtual-reality environment.

Still further, in various embodiments, virtual-reality tag content isfiltered so that certain types of content are presented to specificusers based on their preferences. In this manner, content is screenedfor younger users, or content that is not desired or interesting to eachspecific HMD user is filtered out.

In one embodiment, a method for tagging a virtual item within a virtualenvironment is described. The method includes sending, via a computernetwork, virtual environment data for display of a first virtualenvironment on a first head-mounted display. The first head-mounteddisplay is worn by a first user. The first virtual environment includesa virtual item. The method includes receiving, via the computer networkand a first user account of the first user, an indication of a selectionassociated with the virtual item. The method further includes sending,via the computer network, option data for allowing entry of contentregarding the virtual item upon receiving the indication of theselection. The method includes receiving, via the computer network, thecontent. The method further includes associating the content with thevirtual item and the user account. The method includes sending, via thecomputer network, tagged data for displaying a tag associated with thevirtual item. The tag indicates that the content is associated with thevirtual item.

In an embodiment, a system for tagging a virtual item within a virtualenvironment is described. The system includes a first head-mounteddisplay, which is worn by a first user. The system includes one or moreservers coupled to the first head-mounted display via a computernetwork. The one or more servers send, via the computer network, virtualenvironment data for display of a first virtual environment on the firsthead-mounted display. The first head-mounted display displays the firstvirtual environment. The one or more servers receive, via the computernetwork and a first user account of the first user, an indication of aselection associated with the virtual item displayed on the firsthead-mounted display. The one or more servers further send, via thecomputer network, option data for allowing entry of content regardingthe virtual item upon receiving the indication of the selection. Thefirst head-mounted display displays the option data for the entry of thecontent. The one or more servers further receive, via the computernetwork, the content and associate the content with the virtual item.The one or more servers send, via the computer network, tagged data fordisplaying a tag associated with the virtual item. The tag indicatesthat the content is associated with the virtual item. The firsthead-mounted display displays the tag.

In an embodiment, a system for tagging a virtual item within a virtualenvironment is described. The system includes a head-mounted displaythat displays a first virtual environment. The head-mounted display isworn by a first user. The first virtual environment includes a virtualitem. The system further includes a communication device coupled to thehead-mounted display for sending, via a computer network and a firstuser account of the first user, an indication of a selection associatedwith the virtual item. The communication device receives, via thecomputer network, option data for allowing entry of content regardingthe virtual item after the indication of the selection is sent. Thehead-mounted display displays the content. The communication devicesends the content via the computer network to a server for associatingof the content with the virtual item. The communication device receivestag data for displaying a tag associated with the virtual item. The tagindicates that the content is associated with the virtual item.

Some advantages of the herein described systems and methods includecommunicating information, e.g., content, etc., between various users.The communication of information helps the users in making a betterdetermination whether to buy a real item represented by a virtual item,which is associated with a tag and a comment. Moreover, thecommunication is asynchronous, e.g., when one or both users are notlogged into their corresponding accounts, or synchronous, e.g., whenboth users are logged into their corresponding accounts, etc.

Further advantages of the herein described systems and methods includeproviding a three-dimensional virtual environment in whichthree-dimensional virtual items are associated with content. A real userprovides inputs via hand gestures or via a hand-held controller tochange a perspective with which the three-dimensional virtual items aredisplayed within an HMD. The real user feels as if he/she is in thethree-dimensional virtual environment. The real user provides a commentupon viewing the three-dimensional virtual item and a tag is generatedto indicate that the three-dimensional virtual item is associated withthe comment. When another real user visits the three-dimensional virtualenvironment again, the other real user manipulates the HMD to change aperspective of the three-dimensional virtual item. The other real userselects the tag or the three-dimensional virtual item to view thecomment to determine whether to purchase a real item represented by thethree-dimensional virtual item and/or whether to make further commentson the three-dimensional virtual item. The display of the comment and/orthe tag with the three-dimensional virtual item provides a way topresent information on the HMD so that there is no confusion to theother real user that the three-dimensional virtual item represents thesame real item. This is in comparison to a case where different views ofan item are displayed and different reviews are provided on the item bydifferent users. Some users are confused whether the views are providedfor the same item or to different items, represented by the differentviews.

Additional advantages of the herein described systems and methodsinclude dynamically linking a virtual item to a three-dimensional modelto allow a real user to ascertain that the virtual item represents asingle real item. For example, when the real user views atwo-dimensional view, e.g., a flat floor, a flat closed door, a flatwindow, etc., of a virtual item in a virtual environment on ahead-mounted display, the user selects the virtual item and providescontent to associate with the virtual item. When the content isassociated with the virtual item, a tag is displayed besides the virtualitem. When the virtual item or the tag is accessed by another real useron a head-mounted display, a processor generates a three-dimensionalview of the virtual item based on an orientation of a face of the otherreal user on which the head-mounted display is worn. The other real userprovides additional content to associate with the same virtual item orviews the content provided by the real user to determine whether topurchase a real item associated with the virtual item. There is noconfusion that the virtual item that is associated with the comment, theadditional comment, and the tag is the same.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a system for interactive gameplay of a video game, inaccordance with an embodiment of the disclosure.

FIG. 2A illustrates a head-mounted display (HMD), in accordance with anembodiment of the disclosure.

FIG. 2B illustrates one example of an HMD user interfacing with a clientsystem, and the client system providing content to a second screendisplay, which is referred to as a second screen, in accordance with oneembodiment.

FIG. 3 conceptually illustrates the function of an HMD in conjunctionwith an execution of a video game, in accordance with an embodiment ofthe disclosure.

FIG. 4A is a diagram of an embodiment of a system to illustratecommunication between HMDs and a server for facilitating tagging withina virtual environment.

FIG. 4B is a diagram of an embodiment of a database to illustratelinking of a virtual item with different content and different useridentifiers (IDs).

FIG. 5A is a diagram illustrating a virtual environment that isdisplayed on an HMD that is worn by a user to illustrate generation of atag.

FIG. 5B is a diagram of an embodiment of a virtual environment toillustrate presentation of content when a user selects a tag or avirtual item.

FIG. 5C is a diagram of an embodiment of a virtual environment toillustrate content that is added on top of another content in thevirtual environment.

FIG. 5D is a diagram to illustrate different perspective view of avirtual item.

FIG. 6A is a diagram of an embodiment of a virtual environment toillustrate a placement of the virtual item in a location that is moreprominent than locations of other virtual items within a virtualenvironment.

FIG. 6B is a diagram of an embodiment of a virtual environment toillustrate restriction from access to virtual items by a user whose useraccount is restricted.

FIG. 7A is a diagram of a virtual environment that is displayed on anHMD to illustrate generation of a tag.

FIG. 7B is a diagram of an embodiment of a virtual environment that isdisplayed on an HMD to illustrate presentation of content when a userselects a tag or a virtual item.

FIG. 8 is a diagram of an embodiment of a virtual environment that isdisplayed on an HMD to illustrate a display of a virtual item in aprominent manner.

FIG. 9 is a diagram of an embodiment of a virtual environment that isdisplayed on an HMD to illustrate use of tags during play of a game.

FIG. 10 illustrates components of a head-mounted display, in accordancewith an embodiment of the disclosure.

FIG. 11 is a block diagram of a game system, according to variousembodiments of the disclosure.

DETAILED DESCRIPTION OF THE EMBODIMENTS

FIG. 1 illustrates a system for interactive gameplay of a video game, inaccordance with an embodiment of the disclosure. A user 100 is shownwearing a head-mounted display (HMD) 102. The HMD 102 is worn in amanner similar to glasses, goggles, or a helmet, and is configured todisplay a virtual environment to the user 100. Examples of a virtualenvironment include a virtual reality scene or an augmented realityscene. The virtual environment is of a video game or a shoppingplatform, which users access to view virtual items to determine whetherto purchase real items that are represented by the virtual items. In oneembodiment, the shopping platform is a part of the video game.

The HMD 102 provides a very immersive experience to the user by virtueof its provision of display mechanisms in close proximity to the user'seyes. Thus, the HMD 102 can provide display regions to each of theuser's eyes which occupy large portions or even the entirety of thefield of view of the user. An example of the HMD 102 is described inU.S. patent application Ser. No. 14/270,257, filed on May 5, 2014,titled “METHODS AND SYSTEMS FOR SOCIAL SHARING HEAD MOUNTED DISPLAY(HMD) CONTENT WITH A SECOND SCREEN”, and published as U.S. PatentApplication Publication No. 2015-0243078, which is hereby fullyincorporated by reference herein in its entirety. Another example of theHMD 102 is described in U.S. patent application Ser. No. 14/296,379,filed on Jun. 4, 2014, titled “HEAD MOUNTED DISPLAY”, and published asU.S. Patent Application Publication No. 2014-0361956, which is herebyfully incorporated by reference herein in its entirety.

In one embodiment, the HMD 102 is connected to a computer 106. Theconnection to computer 106 is wired or wireless. Examples of a wiredconnection, as used herein, include a parallel transfer connection, aserial transfer connection, and a universal serial bus (USB) connection.Examples of a wireless connection, as used herein, include a Bluetooth™connection. The computer 106 is any general or special purpose computerknown in the art, including but not limited to, a gaming console,personal computer, laptop, tablet computer, mobile device, cellularphone, tablet, thin client, set-top box, media streaming device, etc. Inone embodiment, the computer 106 executes a video game, and outputsvideo and audio generated from the execution of the video game forrendering on one or more display screens of the HMD 102.

In some embodiments, the user 100 wears and operates one or two gloveinterface objects 104 a to provide input for the video game.Additionally, a camera 108 captures images of a real-world interactiveenvironment in which the user 100 is located. The camera 108 is definedto include one or more image capture devices, such as a stereoscopicpair of cameras, an infrared (IR) camera, a depth camera, or acombination thereof. In an embodiment, the camera 108 is located withinthe computer 106. Examples of the real-world interactive environmentinclude an enclosed space, a room, a warehouse, and a floor of abuilding. These captured images are analyzed to determine a location andmovements of the user 100, the HMD 102, and the glove interface objects104 a. In one embodiment, the glove interface object 104 a includes alight which is tracked or markers that are tracked to determine itslocation and orientation.

As described below, a way the user 100 interfaces with the virtualenvironment displayed on the HMD 102 varies, and other interface devicesin addition to glove interface objects 104 a, are optionally used. Forinstance, single-handed controllers are used, as well as two-handedcontrollers. In some embodiments, the controllers are tracked themselvesby tracking lights associated with the controllers, or tracking ofshapes, sensors, and inertial data associated with the controllers.Using these various types of controllers, or even simply hand gesturesthat are made and captured by one or more cameras, it is possible tointerface, control, maneuver, interact with, and participate in thevirtual environment presented on the HMD 102.

Additionally, in some embodiments, the HMD 102 includes one or morelights which can be tracked to determine a location and orientation ofthe HMD 102. The HMD 102 includes one or more microphones to capturesound from the real-world interactive environment. In some embodiments,the microphones are separate from the HMD 102, e.g., integrated withinthe camera 108, etc. Sound captured by a microphone array is processedto identify a location of a sound source. Sound from an identifiedlocation is selectively utilized or processed to the exclusion of othersounds not from the identified location.

In another embodiment, the computer 106 functions as a thin client incommunication over a computer network 110 with a cloud gaming provider112. Examples of the computer network 110 include a local area networksuch as an Intranet, or a wide area network such as the Internet, or acombination thereof. The cloud gaming provider 112 is a server systemthat includes one or more servers for execution of the video game. In anembodiment, the cloud gaming provider 112 is a virtual machine. Thecloud gaming provider 112 maintains and executes the video game beingplayed by the user 102. The computer 106 transmits inputs from the HMD102, the glove interface objects 104 a, and/or the camera 108, to thecloud gaming provider 112, which processes the inputs to affect a gamestate of the video game. An output from the video game, such as videodata, audio data, and haptic feedback data, is transmitted to thecomputer 106. The computer 106 further processes the output beforetransmission or directly transmits the output to relevant devices. Forexample, video and audio data streams are provided to the HMD 102,whereas a vibration feedback command is provided to the glove interfaceobject 104 a.

In one embodiment, the HMD 102, the glove interface object 104 a, andthe camera 108, are networked devices that connect to the computernetwork 110 to communicate with the cloud gaming provider 112. Forexample, the computer 106 is a local network device, such as a router,that does not otherwise perform video game processing, but whichfacilitates passage of network traffic. The connections between thecomputer network 110 and the HMD 102, glove interface object 104 a, orthe camera 108 are wired or wireless.

Additionally, though embodiments in the present disclosure are describedwith reference to a head-mounted display, it will be appreciated that inother embodiments, non-head mounted displays are used instead, includingwithout limitation, a television, projector, liquid crystal display(LCD) display screen, portable device screen (e.g. tablet, smartphone,laptop, etc.) or any other type of display that render video and/orprovide for display of the virtual environment in accordance with thepresent embodiments.

FIG. 2A illustrates a head-mounted display (HMD), in accordance with anembodiment of the disclosure. As shown, the HMD 102 includes a pluralityof lights 200A-H. Each of these lights 200A-H have specific shapes, andhave the same or different colors. The lights 200A, 200B, 200C, and 200Dare arranged on a front surface of the HMD 102. The lights 200E and 200Fare arranged on a side surface of the HMD 102. And the lights 200G and200H are arranged at corners of the HMD 102 to span the front surfaceand a side surface of the HMD 102. It will be appreciated that thelights are identified in captured images of an interactive environmentin which a user uses the HMD 102. Based on identification and trackingof the lights, the location and orientation of the HMD 102 in thereal-world interactive environment is determined. It will further beappreciated that some of the lights are or are not visible dependingupon the particular orientation of the HMD 102 relative to the camera108 of FIG. 1. Also, different portions of lights (e.g. lights 200G and200H) are exposed for image capture depending upon the orientation ofthe HMD 102 relative to the camera 108.

In one embodiment, the lights indicate a current status of the HMD 102to other users in the real-world interactive environment. For example,some or all of the lights 200A-H have a certain color arrangement,intensity arrangement, blink, have a certain on/off configuration, orhave another arrangement indicating a current status of the HMD 102. Byway of example, the lights 200A-H display different configurationsduring active gameplay of the video game versus other non-activegameplay aspects of the video game, such as navigating menu interfacesor configuring game settings, during which a gameplay virtual scene maybe inactive or paused. The lights 200A-H indicate relative intensitylevels of gameplay. For example, an intensity of the lights 200A-H or arate of blinking of the lights 200A-H increase when the intensity of thegameplay increases. In this manner, a person standing near the userviews one or more of the lights 200A-H on the HMD 102 and understandsthat the user is actively engaged in intense gameplay, and may not wishto be disturbed at that moment.

The HMD 102 additionally includes one or more microphones. In theillustrated embodiment, the HMD 102 includes microphones 204A and 204Bdefined on the front surface of the HMD 102, and microphone 204C definedon a side surface of the HMD 102. By utilizing an array of microphones,sound from each of the microphones is processed by the computer 106 orthe cloud gaming provider 112 to determine a location of the sound'ssource. The location of the sound's source is utilized in various ways,including exclusion of unwanted sound sources, association of a soundsource with a visual identification, etc.

The HMD 102 includes one or more image capture devices. In theillustrated embodiment, the HMD 102 is shown to include image capturedevices 202A and 202B. By utilizing a stereoscopic pair of image capturedevices, three-dimensional (3D) images and video of the real-worldinteractive environment are captured from a perspective of the HMD 102.Such video is presented to the user to provide the user with a “videosee-through” ability while wearing the HMD 102. That is, though the usercannot see through the HMD 102 in a strict sense, the video captured bythe image capture devices 202A and 202B (e.g., or one or more frontfacing cameras 108′ disposed on the outside body of the HMD 102, asshown in FIG. 3 below) nonetheless provide a functional equivalent ofbeing able to see the real-world interactive environment external to theHMD 102 as if looking through the HMD 102. Such video is augmented withvirtual items to provide an augmented reality experience, or is combinedor blended with the virtual items in other ways. Though in theillustrated embodiment, two cameras are shown on the front surface ofthe HMD 102, it will be appreciated that there may be any number ofexternally facing cameras installed on the HMD 102, oriented in anydirection. For example, in another embodiment, there are cameras mountedon the sides of the HMD 102 to provide additional panoramic imagecapture of the environment.

FIG. 2B illustrates one example of an HMD 102 interfacing with thecomputer 106, and the computer 106 providing content to a second screendisplay. The second screen display is referred to herein as a secondscreen 107. As will be described below, the computer 106 includesintegrated electronics for processing the sharing of content from theHMD 102 to the second screen 107. Other embodiments include a separatedevice, module, connector, that will interface between the computer 106and each of the HMD 102 and the second screen 107. In this generalexample, the user 100 is wearing the HMD 102 and is playing a video gameusing a hand-held controller 104. The interactive play by the user 100will produce video game content (VGC), which is displayed interactivelyto the HMD 102.

In one embodiment, the content being displayed on the HMD 102 is sharedto the second screen 107. In one example, a person viewing the secondscreen 107 views the content being played interactively in the HMD 102by the user 100. In another embodiment, another user (e.g. player 2)interacts with the computer 106 to produce second screen content (SSC).The SSC is produced when the player 2 also interacts with the controller104 (or any type of user interface, gesture, voice, or input. The SSC isdisplayed on the second screen 107 along with the VGC received from theHMD 102.

Accordingly, the interactivity by other users who are co-located orremote from an HMD user is social, interactive, and more immersive toboth the HMD user and the other users who are viewing the content playedby the HMD user on the second screen 107. A user who is remote the HMDuser 100 is labeled as 130. As illustrated, the computer 106 isconnected to the computer network 110. The computer network 110 alsoprovides access to the computer 106 to content from various contentsources 120. The content sources 120 includes any type of content thatis accessible over the computer network 110.

Such content, without limitation, includes video content, movie content,streaming content, social media content, news content, encoded streamingcontent, friend content, advertisement content, etc. In one embodiment,the computer 106 is used to simultaneously process content for an HMDuser, such that the HMD 102 is provided with multimedia contentassociated with interactivity of the user 100 during gameplay. Thecomputer 106 also provides other content, which may be unrelated to thevideo game content, to the second screen. The computer 106, in oneembodiment, receives the second screen content from one of the contentsources 120, or from a local user, or a remote user.

FIG. 3 conceptually illustrates a functionality of the HMD 102 inconjunction with the video game, in accordance with an embodiment of thedisclosure. The video game is defined by a game engine 320 whichreceives inputs to update a game state of the video game. The game stateof the video game is defined, at least in part, by inputs received fromgestures or voice commands of the user 100. The inputs are captured bythe camera 108, the HMD 102, and/or by sensors, such as accelerometersand gyroscopes. The sensors are integrated within the HMD 102 or withinglove interface objects 104 a or within hand-held controllers. Examplesof the game state include various aspects of the current gameplay, suchas the presence and location of the virtual objects, conditions of thevirtual environment, triggering of events, user profiles, and viewperspectives, etc.

In the illustrated embodiment, the game engine 320 receives, by way ofexample, controller input 314, audio input 316 and motion input 318. Thecontroller input 314 is defined from the operation of a gamingcontroller separate from the HMD 102, such as the handheld controller(e.g. Sony DUALSHOCK®4 wireless controller, Sony PlayStation®Move motioncontroller) or the glove interface object 104 a. By way of example, thecontroller input 314 includes directional inputs, button presses,trigger activation, movements, gestures, or other kinds of inputsprocessed from the operation of the gaming controller. The audio input316 is received by the game engine 320 from a sound of the user 100 thatis captured by a microphone 302 of the HMD 102, or from a microphoneincluded in the image capture device 108, or elsewhere in the real-worldinteractive environment. Each of the microphones 204A through 204C is anexample of the microphone 302. The motion input 318 is received by thegame engine 320 from a motion sensor 300 included in the HMD 102 and/orfrom the camera 108 as it captures images of the HMD 102. The gameengine 320 receives the inputs, e.g., the controller input 314, theaudio input 316 and the motion input 318, etc., and processes the inputsaccording to a configuration of the game engine 320 to update the gamestate of the video game. The game engine 320 outputs game state data tovarious rendering modules which process the game state data to definecontent which will be presented to the user 100 or to another user viaanother HMD. Examples of the content include video content and audiocontent.

In the illustrated embodiment, a video rendering module 322 is definedto render a video stream for presentation on the HMD 102 based on thegame state data. The video stream is presented by a display/projectormechanism 310, and viewed through optics 308, such as one or morelenses, by an eye 306 of the user 100. An audio rendering module 304renders an audio stream for listening by a user. In one embodiment, theaudio stream is output through a speaker 304 associated with the HMD102. It should be appreciated that speaker 304 takes the form of an openair speaker, headphones, or any other kind of speaker capable ofpresenting audio.

In an embodiment, the sound that is captured as the audio input 316 isstored in a memory device of the computer 106 or of the cloud gamingservice 112 in the form of audio data. As used herein, a memory deviceis a read-only memory device (ROM), or a random access memory (RAM), ora combination thereof. The audio data is accessed from the computer 106or of the cloud gaming service 112 to be output as the audio stream onan HMD, which is other than the HMD 102.

In one embodiment, a gaze tracking camera 312 is included in the HMD 102to enable tracking of a gaze of the user 100. The gaze is an example ofa gesture performed by the user 100. The gaze tracking camera capturesimages of eyes of the user 100. The images are analyzed to determine agaze direction of the user 100. In one embodiment, information about thegaze direction of the user 100 is utilized to affect the videorendering. For example, if a the eyes of the user 100 are determined tobe looking in a specific direction, then the video rendering for thatdirection is prioritized or emphasized, such as by providing greaterdetail or faster updates in the region where the user 100 is looking. Itshould be appreciated that the gaze direction of the user 100 is definedrelative to the HMD 102, relative to the real-world interactiveenvironment in which the user 100 is situated, and/or relative to thevirtual environment that is being rendered on the HMD 102.

Additionally, a tactile feedback module 326 provides signals to tactilefeedback hardware included in either the HMD 102 or another deviceoperated by the user 100, such as the hand-held controller 104. Thetactile feedback module 326 takes the form of various kinds of tactilesensations, such as vibration feedback, temperature feedback, pressurefeedback, etc.

At present, streaming services for sharing game replays are verypopular. The DualShock®4 wireless controller includes a “share button”directly on the controller to enable such sharing. Implementations ofthe present disclosure improve sharing replays for people who wish toexplore replays using an HMD. Implementations of the present disclosureprovide for rendering of a game replay with a very wide field of view toallow a spectator to move his head freely using the HMD 102 and view thereplay from novel vantage points. The traditional streaming approachwould limit the replay to only what the original player viewed, so thatthe view direction would be independent of the spectator's head positionand orientation, and if the spectator using the HMD 102 moved his head,nothing would change.

Implementations of the disclosure provide for the rendering of videos ina wide enough field of view to support novel viewpoints in the HMD 102.A custom build of a game engine that runs on a cloud server (e.g. onconsole gaming hardware, e.g. PlayStation®4 hardware, in the cloud),that accepts as input game state streamed from the original player'sgame engine and uses it to render an extremely wide field of view (e.g.150 degree plus) view of the game, that is then used for real-timestreaming and/or pre-recorded playback of that game session. It will beappreciated that the extremely wide field of view is in excess of theHMD 102's field of view, allowing for the spectator wearing the HMD 102to look around in the replay. The actual game is configured to streamits state to the networked version of the engine.

As described above, there is a need to provide users the ability tospectate, e.g., watch the interactive activity being experienced byusers wearing HMDs 102. For example, an HMD player is immersed in theactivity presented in the HMD 102, while other persons are co-locatedwith the player. These other co-located players find enjoyment inwatching the interactivity experienced or virtual reality scene beingviewed by the player via the HMD 102. As used herein, the HMD player isone that is viewing content presented on the HMD 102, or is one that isinteracting with some content resented on the HMD 102, or is playing agame presented on the HMD 102. As such, reference to the HMD player, ismade with reference to a user that is wearing the HMD 102, irrespectiveof the type of content being presented on the HMD 102.

In still other embodiments, other persons that are not co-located withthe HMD player wish to view the content, interactivity, or media beingpresented in the HMD 102 worn by the HMD player. For instance, a websiteis provided to present users with the ability to select from differentHMD players, so as to watch and spectate while the HMD player performshis or her activities. This example is similar to standard Twitch-typeexperiences, which allow users connected to the computer network 110 toaccess a website and search for different types of content or mediabeing played by remote players. The remote players, in some embodiments,are playing games using an HMD, such as the HMD 102.

In other embodiments, the remote players are playing games or watchingcontent using a display screen of a device or a television displayscreen. Broadly speaking, users wishing to watch the activity of the HMDplayer that is remote, e.g., over a website, select specific players ortypes of games, or thumbnails of the games, or thumbnails of thecontent, to view the activity being directed by the HMD player. Thus, awebsite is provided that enables users to view and select specificinteractive content that may be actively played by the HMD player. Theremote viewer wishing to view the activity by the HMD player, simplyclicks on that content and begins watching.

The person watching and viewing the actions by the HMD player isgenerally referred to as a spectator. Spectators are those persons whoare given access to view the activities, interactivities, actions,movements, etc., but are not necessarily controlling the game action.For this reason, these viewers are referred to as spectators. In thecontext of the HMD player, the content being presented on the display ofthe HMD 102 is dynamic and is controlled by the movements of the HMDplayer. For example, when the HMD player moves his or her head around,the HMD player is presented with different content that is viewable,similar to the way real world viewing of a person's surroundings canoccur.

In one embodiment, each of the game engine 320, the video renderingmodule 322 and the audio rendering module 324 is hardware, such as anintegrated circuit, or a computer software program, that is executed byone or more processors. In an embodiment, each of the game engine 320,the video rendering module 322 and the audio rendering module 324 isexecuted by one or more processors of the cloud gaming service 112 or ofthe computer 106. As used herein, a processor is an application specificintegrated circuit (ASIC), or a programmable logic device (PLD), or acentral processing unit (CPU).

FIG. 4A is a diagram of an embodiment of a system 400 to illustratecommunication between users 1 through 4 and a server 402 forfacilitating tagging within a virtual environment. The system 400includes the server 402, the computer network 110, and multiple HMD s404A, 404B, 404C, and 404D. Any of the HMDs 404A, 404B, 404C, and 404Dis an example of the HMD 102 (FIG. 1). Moreover, the server 402 is apart of the cloud gaming provider 112. Also, in one embodiment, theterms user 1 and user 102 are used interchangeably herein. The HMD 404Ais worn by the user 1 on his/her head. Similarly, the HMD 404B is wornby the user 2 on his/her head, the HMD 404C is worn by the user 3 onhis/her head, and the HMD 404D is worn by the user 4 on his/her head.

The server 402 includes a processor 406, a network interface controller(NIC) 408, and a memory device 410. The processor 406 includes a ratingdeterminator 412, a content amount calculator 414, and a permissioncalculator 416, all of which are further described below. Each of therating determinator 412, the content amount calculator 414, and thepermission calculator 416 is a software module or a hardware module. Anexample of the software module includes a computer program code andexamples of the hardware module include an ASIC and a PLD. As usedherein, the processor 406 is a CPU, or a controller, or an ASIC, or aPLD. Examples of the NIC 408 include a network interface card that isused to connect the server 402 to the computer network 110. Moreover,examples of the memory device 410 include a ROM, a RAM, or a combinationthat includes RAM and ROM. To illustrate, the memory device 410 is aflash memory device or a redundant array of independent disks (RAID).The processor 406 is coupled to the NIC 408 and the memory device 410.

The memory device 410 stores a user account 1 that is assigned to theuser 1, a user account 2 that is assigned to the user 2, a user account3 that is assigned to the user 3, and a user account 4 that is assignedto the user 4. In some embodiments, a user account is assigned to a userby the processor 406 when the user provides login information forlogging into the user account. Examples of the login information includea user name, user information, a user identifier (ID), an email address,a password, or a combination thereof. The processor 406 determineswhether the login information is assigned to another user, e.g., otherthan the user 1, and if not, the processor 406 links the logininformation to the user account of the user 1 to assign the user accountto the user 1.

The processor 406 associates, such as links, establishes a one-to-onecorrespondence with, or maps, the user account 1 with a permission set1, associates the user account 2 with a permission set 2, associates theuser account 3 with a permission set 3, and associates the user account4 with a permission set 4. For example, a user ID assigned to the useraccount 1 is linked to the permission set 1, a user ID assigned to theuser account 2 is linked to the permission set 2, a user ID assigned tothe user account 3 is linked to the permission set 3, and a user IDassigned to the user account 4 is linked to the permission set 4.

In one embodiment, a permission set associated with a user account of auser includes preferences received from the user via the user account,or privacy settings received from the user via the user account, orrestrictions placed on the user account by the processor 406 based onthe user's browsing history, or a demographic of the user, orrestrictions placed on the user account by another user via anadministrator user account, or a combination thereof. For example, auser logs into his/her user account to indicate a privacy setting inwhich no users other than social network friends of the user are able toview information posted by the user via the user account. As anotherexample, based on the demographic, e.g., age, gender, etc., of the userreceived via a user account of the user, the processor 406 determinesthat the user cannot access information posted to the user account. Asyet another example, when the user 1 indicates via his/her user account1 that the user account 2 cannot access content created by the user 1,the permission set 2 of the user 2 is affected and is not allowed toaccess the content. The browsing history indicates which virtual item isinteresting to a user or is desired by the user.

It should be noted that in some embodiments, a user logs into his/heruser account when the processor 406 of the server 402 authenticates thelogin information received via the computer network 110 from an HMD wornby the user. The user selects various alphanumeric characters displayedon the HMD via the hand-held controller 104 operated by the user toprovide the login information. In some embodiments, the logininformation is received from the HMD via the computer 106 (FIG. 1) andthe computer network 110. For example, the user uses the hand-heldcontroller 104 to provide the login information to the computer 106. Thehand-held controller 104 communicates the login information to thecomputer 106 via a wired or wireless connection, examples of which areprovided above.

The memory device 410 further stores an association, such as a link, aone-to-one correspondence, or a mapping, between a rating assigned to avirtual item by the rating determinator 412 and the virtual item. Forexample, a virtual item 502A is assigned a rating 1 and a virtual item502B is assigned a different rating 2. In some embodiments, a rating isassigned to the virtual item by the rating determinator 412. Examples ofa rating include a number, or an alphabet, or a combination thereof. Invarious embodiments, the terms rating and ranking are usedinterchangeably herein. A rating is assigned to a virtual item based oninformation regarding the virtual item, e.g., comments, posts, ranking,opinions, interests, hobbies, etc., posted within one or more useraccounts of one or more users. For example, the virtual item 502A isassigned a higher rating than the virtual item 502B based on a number ofuser accounts expressing interest in the virtual items 502A and 502B. Toillustrate, a first number of user accounts indicate that the virtualitem 502A is of interest. Users access the first number of user accountsvia hand-held controllers to indicate that that the virtual item 502A isof interest to them. Also, a second number of user accounts indicatethat the virtual item 502B is interesting. Users access the secondnumber of user accounts via hand-held controllers to indicate that thatthe virtual item 502B is of interest to them. When the first number ofuser accounts is greater than the second number of user accounts, thevirtual item 502A is assigned a higher rating than the virtual item502B. As another example, when the content amount calculator 414determines that a number of users that interact with, e.g., hover over,click on, etc., the virtual item 502A via their corresponding useraccounts to indicate their interest is greater than a number of usersthat interact with the virtual item via their corresponding useraccounts, the virtual item 502A is assigned a higher rating by therating determinator 412 than the virtual item 502B. As yet anotherexample, when the content amount calculator 414 determines that a numberof comments provided to the virtual item 502A via their correspondinguser accounts regarding the virtual item 502A in a virtual environmentis greater than a pre-determined number of comments and that a number ofcomments provided to other virtual items in the virtual environment isless than the pre-determined number, the virtual item 502A is assigned ahigher rating by the rating determinator 412 than the other virtualitems. The number of comments is an example of an amount of content. Forexample, content 506, illustrated below in FIG. 5B, is one comment andanother content is another comment. It should be noted that thepre-determined number corresponds to, e.g., has a one-to-onerelationship with, is linked to, etc., a pre-determined rating.

In some embodiments, instead of the processor 406, multiple processorsare used to perform the functions illustrated herein. In variousembodiments, instead of the memory device 410, multiple memory devicesare used to store the data described herein.

In various embodiments, functions described herein as being performed bythe server 402 are performed by multiple servers of the cloud gamingservice 112.

FIG. 4B is a diagram of an embodiment of a system 450 to illustrate adatabase 452 that stores an association between virtual items, content,and user IDs. The system 450 includes the processor 460 and the memorydevice 410. The processor 460 includes an associator module 454 thatcreates associations, e.g., links, mappings, etc., among the virtualitems, content, and user IDs. For example, the associator module 454links the virtual item 502A to a content IDa1, a content IDa2, and acontent IDa3. A content ID is created by the processor 460 to identifycontent. For example, the content IDa1 is created by the processor 460to identify the content 506. Similarly, the content IDa2 is created bythe processor 460 to identify additional content 550, which is describedbelow in FIG. 5C. Examples of user IDs includes a combination ofalphanumeric characters or a combination of alphanumeric characters andsymbols. Examples of content IDs includes a combination of alphanumericcharacters or a combination of alphanumeric characters and symbols. Thecontent 506 is created by the user 1 and received via the user account 1that is assigned a user ID1. Moreover, the additional content 550 iscreated by the user 2 and received via the user account 2 that isassigned a user ID2. Also, content having the content IDa3 is created bythe user 3 and received via the user account 3 that is assigned a userID3. As another example, the associator module 454 links the virtualitem 502B to a content IDb1 and content IDb2. Moreover, content havingthe content IDb1 is created by the user 1 and received via the useraccount 1 that is assigned the user ID1. Also, content having thecontent IDb2 is created by the user 2 and received via the user account2 that is assigned the user ID2.

FIG. 5A is a diagram illustrating a virtual environment A1 that isdisplayed on the HMD 404A that is worn by the user 1. The virtualenvironment A1 is a view of a virtual store for leasing or buyingvirtual items 502A, 502B, 502C, 502D, and 502E. For example, upondetermining that the user account 1 of the user 1 is authenticated anddetermining that the user account 1 has permission to access the virtualstore, a computer program is executed by the processor 406 (FIG. 4A) togenerate data, which is encoded by an encoder of the server 402, andsent by the NIC 408 of the server 401 via the computer network 110 fordisplaying the virtual environment A1 on the HMD 404A. The encoder ofthe server system 402 is coupled to the processor 406 and to the NIC408. The NIC 408 applies a communication protocol, e.g., TransmissionControl Protocol over Internet Protocol (TCP/IP), etc., to packetize theencoded data to generate packets, which are then sent via the computernetwork 110. The computer 106 includes the NIC 107 that depacketizes theencoded data, and further includes a decoder that decodes the encodeddata. The computer 106 sends the decoded data via the wired or wirelessconnection to the HMD 404A for display of the decoded data on thedisplay device of the HMD 404A. In one embodiment, the HMD 404A includesa NIC that applies the communication protocol to extract the encodeddata and further includes a decoder that decodes the encoded data.

The user 1 has logged into his/her user account 1 to access the virtualenvironment A1. After logging into the user account 1, the user 1 makesa selection associated with the virtual item 502A. For example, the user1 makes a gesture, e.g., a pointing gesture to point his/her finger atthe virtual item 502A, etc., to select the virtual item 502A. As anotherexample, the user 1 hovers a mouse pointer on the virtual item 502A orclicks on the virtual item 502 a with the mouse pointer to select thevirtual item 502A. A direction of movement of the mouse pointer iscontrolled via a hand-held controller 520A, e.g., a Playstation Move™controller, a joystick, a pen controller, a wireless controller, etc.,operated by the user 1. The hand-held controller 520A is an example ofthe hand-held controller 104 of FIG. 2B.

To indicate that the virtual item 502A is about to be selected, theprocessor 406 generates a virtual curve VC1 that is displayed in the HMD404A as pointing to the virtual item 502A. Data for the display of thevirtual curve VC1 is packetized and sent from the NIC 408 via thecomputer network 110 to the NIC 107 of the computer 106. The NIC 107 ofthe computer 106 depacketizes the data and sends the data via the wiredor wireless connection to the HMD 404A to display the virtual curve VC1on the display device of the HMD 404A. In one embodiment, the data forthe display of the virtual curve VC1 is packetized and sent from the NIC408 via the computer network 110 to the NIC of the HMD 404A. The NIC ofthe HMD 404A depacketizes the data to display the virtual curve VC1 onthe display device of the HMD 404A.

The gesture made by the user 1 is captured by the camera 108 (FIG. 1),or a camera of the HMD 404A. The camera 108 sends an indication of,e.g., data indicating, image data of, etc., the gesture via a wired or awireless connection to the computer 106 to packetize the image data andthe packetized data is sent from via the computer network 110 to the NIC408. When the camera of the HMD 404A captures the gesture, the camera ofthe HMD 404A sends the indication of the gesture via the wired orwireless connection to the computer 106, which packetizes and sends theindication of the gesture via the computer network 110 to the NIC 408 ofthe server 402. In one embodiment, when the camera of the HMD 404Acaptures the gesture, the HMD 404A packetizes and sends the indicationof the gesture via the computer network 110 to the NIC 408 of the server402. Similarly, when the selection of the virtual item 502A is made viathe mouse pointer, data indication the selection is sent from thehand-held controller 520A via the wired or wireless connection to thecomputer 106. The computer 106 packetizes the data indicating theselection and sends the packets via the computer network 106 to the NIC408.

The NIC 408 of the server 402 receives the data indicating the selectionmade using the gesture or the click and provides the data to theprocessor 406 of the server 402. The processor 406 processes the dataindicating the selection associated with the virtual item 502A by theuser 1 to generate option data, e.g., data for generating a field forentry of text, data for enabling entry of audio content, data forenabling receipt of video content, a button for enabling entry of theaudio content, a button for enabling recording of the video content,etc., and provides the option data to the NIC 408. The NIC 408packetizes and sends the option data via the computer network 110 to theNIC 107 of the computer 106, which sends the option data via the wiredor wireless connection to the HMD 404A. In one embodiment, the NIC 408packetizes and sends the option data via the computer network 110 to theNIC of the HMD 404A.

The option data is displayed as an option 507, e.g., an audio button, avideo recording button, a text entry field, etc., on the HMD 404A forentry of content 506, which is illustrated in FIG. 5B. For example, afield is displayed within the virtual environment A1. The user 1 makes agesture to select the text entry field and provides the content 506,e.g., text, graphics, etc. As another example, the user 1 clicks thehand-held controller 520A to select the text entry field and furtherselects various alphanumeric characters displayed on the HMD 404A toprovide the content 506. As yet another example, the user 1 provides thecontent 506, such as sound data, into a microphone of the HMD 404A. Asstill another example, the user 1 provides the content 506, such asvideo content, to the camera 108 or the camera on the HMD 404A. Examplesof the content 506 include text, audio, video, graphics, or acombination thereof. Illustrations of the content 506 include “I lovethis purse”, “this purse is awesome”, “I do not like this purse”, “Thispurse is horrible”, “This purse broke the day after I bought it”, a sademoji, a happy emoji, an upset emoji, a mad emoji, etc.

The content 506 is sent from the HMD 404A via the wired or wirelessconnection to the computer 106. The NIC 107 of the computer 106packetizes the content 506 and sends the packets via the computernetwork 110 to the NIC 408. The NIC 408 receives the content 506 andprovides the content 506 to the processor 406. In one embodiment, thecontent 506 is packetized by and sent from the NIC of the HMD 404A viathe computer network 110 to the NIC 408.

The processor 406 of the server 402 associates, such as establishes aone-to-one correspondence between, establishes a mapping between, a linkbetween, or establishes a unique relationship between, the virtual item502A, the content 506, and the user account 1 to which access isprovided to receive the content 506 from the user 1. For example, withreference to FIG. 4B, the processor 406 creates an entry 461 in thedatabase 452. The entry 461 includes storage in the database 452 of avirtual item IDv1 assigned to the virtual item 502A by the processor406. The entry 461 further includes storage in the database 452 of thecontent IDa1 assigned to the content 506 by the processor 406. Also, theentry 461 includes storage in the database 452 of the user ID1. The userID1 is assigned to the user account 1 by the processor 406. For example,the user ID1 includes a user name, or a password, or a combinationthereof. The user ID1 is provided by the user 1 to authenticate the useraccount 1. The processor 406 establishes a link 455 between the virtualitem IDv1 and the content IDa1 and creates a link 457 between thecontent IDa1 and the user ID1. To illustrate, the processor 406 createsa pointer from a memory region, e.g., a memory cell, of the memorydevice 410 that stores the virtual item IDv1 to a memory region of thememory device 410 that stores the content IDa1. Moreover, the processor406 creates a pointer from a memory region of the memory device 410 thatstores the user ID1 to a memory region of the memory device 410 thatstores the content IDa1. As another example, the processor 406 creates alink 459 between the virtual item IDv1 and the user ID1. To illustrate,the processor 406 creates a pointer from a memory region of the memorydevice 410 that stores the user ID1 to a memory region of the memorydevice 410 that stores the virtual item IDv1.

The processor 406 of the server 402 generates tag data in response toassociating the content 506 with the virtual item 502A and/or with theuser account 1, and sends the tag data to the NIC 408. The NIC 408packetizes the tag data and sends the packetized tag data via thecomputer network 110 to the NIC 107 of the computer 106. The computer106 sends the tag data via the wired or wireless connection to the HMD404A. In one embodiment, the NIC 408 packetizes the tag data and sendsthe packetized tag data via the computer network 110 to the NIC of theHMD 404A.

Referring back to FIG. 4A, the processor 406 of the server 402 creates alink between a tag identifier TI1 that identifies a tag 508A having thetag data and the virtual item IDv1 and creates a link between the tagidentifier TI1 and the content IDa1, and stores the links in thedatabase 452. The tag identifier TI1 includes one or more alphanumericcharacters, or one or more symbols, or a combination thereof, and isassigned to the tag 508A by the processor 406. For example, theprocessor 406 creates a pointer from a memory region of the memorydevice 410 that stores the content IDa1 to a memory region of the memorydevice 410 that stores the tag identifier TI1.

Upon receiving the tag data, the tag 508A is displayed within thevirtual environment A1 on the HMD 404A as being associated with thevirtual item 502A. For example, a pointer of the tag 508A points towardsthe virtual item 502A. As another example, the tag 508A is locatedwithin a pre-determined distance from the virtual item 502A. As yetanother example, the tag 508A is located closest to the virtual item502A compared to all other virtual items in the virtual environment A1.Similarly, a tag 508B associated with a virtual item 508D is generatedand displayed on the HMD 404A.

In some embodiments, a virtual item is a virtual good, e.g., a shoe, adress, a short, a pant, a piece of jewelry, a piece of clothing, avehicle, a building, a house, etc., that is either sold or leased. Forexample, an HMD worn by a user displays an avatar of the user, e.g., avirtual user, etc., wearing the virtual good so that the user candetermine whether to purchase or lease an actual good represented by thevirtual good. In various embodiments, a virtual item is a virtualservice.

FIG. 5B is a diagram of an embodiment of a virtual environment A2 toillustrate presentation of the content 506 when the user 2 selects thetag 508A or the virtual item 502A. One or more buttons on a hand-heldcontroller 520B are selected by the user 2 to provide the logininformation regarding the user account 2 via the wired or wireless linkto the computer 106. The NIC 107 of the computer 106 packetizes andsends the login information of the user account 2 via the computernetwork 110 to the processor 406 of the server 402 for authentication.In one embodiment, a NIC of the HMD 404B packetizes and sends the logininformation of the user account 2 via the computer network 110 to theprocessor 406 of the server 402 for authentication.

When the user information, e.g., the user ID2, that is assigned to theuser account 2 of the user 2 and received from the computer 106 or theNIC of the HMD 404B is authenticated by the processor 406 of the server402, the NIC 408 of the server 402 sends, via the computer network 110,virtual environment data for display of the virtual environment A2 onthe head-mounted display 404B. For example, the virtual environment datafor the virtual environment A2 is encoded by the encoder of the server402 and send to the NIC 408. The NIC 408 sends the virtual environmentdata for display of the virtual environment A2 via the computer network110 to the NIC 107 of the computer 106. The NIC 107 of the computer 106depacketizes the encoded virtual environment data to extract the encodedvirtual environment data. The encoded virtual environment data isdecoded by the decoder of the computer 106 to output the virtualenvironment data and the virtual environment data is sent via a wired orwireless connection from the computer 106 to the HMD 404B for display ofthe virtual environment A2 on a display device of the HMD 404B

In one embodiment, the NIC 408 packetizes the encoded virtualenvironment data and sends the packetized virtual environment data viathe computer network 110 to the HMD 404B. The NIC of the HMD 404Bdepacketizes the encoded virtual environment data to extract the encodedvirtual environment data. The encoded virtual environment data isdecoded by a decoder of the HMD 404B to output the virtual environmentdata for display of the virtual environment A2 on a display device ofthe HMD 404B.

The virtual environment A2 excludes the content 506 and the virtual item502A but includes the tag 508A. In one embodiment, the virtualenvironment A2 includes the virtual item 502A and the tag 508A, butexcludes the content 506. The virtual environment A2 has the othervirtual items 502B, 502C, 502D, and 502E, and the tag 508B. In oneembodiment, the virtual items 502A through 502E and the content 506 arestored in the database 452, and accessible from the database 452 via thecomputer network 110.

The user 2 selects the tag 508A that is associated with the virtual item502A or selects the virtual item 502A. For example, the user 2 makes agesture, which is captured by the camera 108 or by a camera of the HMD404B, to select the tag 508A or the virtual item 502A. To illustrate,the user 2 points his/her finger at the tag 508A to select the tag 508Aor at the virtual item 502A to select the virtual item 502A. To indicatethat the virtual item 502A or the tag 508A is about to be selected, theprocessor 406 generates a virtual curve VC2 that is displayed in the HMD404B as pointing to the virtual item 502A or the tag 508A. As anotherexample, the user 2 selects a button on the hand-held controller 520Boperated by the user 2 to select the tag 508A or the virtual item 502A.When the tag 508A or the virtual item 502A is selected, an indication,e.g., a signal indicating the selection, etc., is sent from the HMD 404Bvia the NIC 107 of the computer 106, the computer network 110, and theNIC 408 to the processor 406 of the server 402. In one embodiment, theindication of the selection of the tag 508A is packetized by the NIC ofthe HMD 404B and sent from the NIC of the HMD 404B via the computernetwork 110 and the NIC 408 to the processor 406 of the server 402.

The permission calculator 416, shown in FIG. 4A, of the processor 406 ofthe server 402 determines whether the user account 2 of the user 2 haspermission to access the content 506 associated with the tag 508A uponreceiving the indication of the selection of the tag 508A or the virtualitem 502A. For example, the processor 406 accesses the permission set 2associated with the user account 2 to determine whether the user account2 has the permission to access the content 506. To illustrate, the useraccount 2 has permission, as indicated by the permission set 2, when theuser 1 provides permission via his/her permission set 1 to allow accessto the content 506. As another illustration, the user account 2 haspermission to access the content 506 when the user account 2 indicatesthat the user 2 satisfies a demographic threshold, e.g., apre-determined age, a pre-determined interest, a pre-determined hobby,etc., indicated in the permission set 2. Upon receiving thedetermination from the permission calculator 416 of the processor 406that the user account 2 has the permission, the NIC 408 of the server402 sends content data via the computer network 110 to the computer 106.The computer 106 sends via the wired or wireless connection the contentdata to the HMD 404B for displaying the content 506 within the useraccount 2 on the HMD 404B. In one embodiment, the content data is sentfrom the NIC 408 via the computer network 110 to the HMD 404B fordisplaying the content 506 via the user account 2 on the HMD 404B.

It should be noted that in an embodiment, when the user 2 logs into theuser account 2 to access the virtual environment A2, the user 1 haslogged out of the user account 1. For example, the user 1 logs out ofthe user account 1 by selecting one or more buttons on the hand-heldcontroller 520A to select a logout button displayed within the virtualenvironment A1 on the HMD 404A. The selection data indicating theselection of the logout button is sent from the HMD 404A via the wiredor wireless connection to the computer 106. The selection data ispacketized and sent from the NIC 107 of the computer 106 via thecomputer network 110 to the NIC 408. In one embodiment, the selectiondata indicating the selection of the logout button is packetized andsent from the NIC of the HMD 404B via the computer network 110 to theNIC 408, which depacketizes the selection data.

The processor 408 of the server 402 receives the selection data from theNIC 408 of the server 402 and determines not to send the virtualenvironment A1 to the HMD 404A via the computer network 110 tofacilitate the logout from the user account 1. As another example of thelogout from the user account 1, the user 1 makes a gesture to select thelogout button displayed within the virtual environment A1. The gestureindicating the selection of the logout button is captured as image databy the camera 108 or the camera of the HMD 404A. The image dataindicating the selection of the logout button is sent from the HMD 404Avia the wired or wireless connection to the computer 106. The image datais packetized and sent from the NIC 107 of the computer 106 via thecomputer network 110 to the NIC 408. In one embodiment, the image dataindicating the selection of the logout button is packetized and sentfrom the NIC of the HMD 404B via the computer network 110 to the NIC408. It should be noted that when the user 1 logs out of the useraccount 1 and the user 2 is logged into the user account 2, there isasynchronous communication between the users 1 and 2 of the content 506.

In one embodiment, the virtual environment A2 includes a search field522 but excludes the content 506, the tag 508A, and the virtual item502A. The user 2, after logging to the user account 2, makes a gestureto search for the virtual item 502A. For example, the user 2 moveshis/her finger to spell a name of the virtual item 502A. When the user 2makes the gesture, the camera 108 or the camera of the HMD 404B worn bythe user 2 captures image data of the gesture. The image data is sentfrom the camera 108 to the computer 106 via the wired or wirelessconnection. The NIC 107 of the computer 106 packetizes the image dataand sends the packets via the computer network 110 to the NIC 408. Inone embodiment, the image data is packetized and sent from the HMD 404Bvia the computer network 110 to the NIC 408 without using the computer106.

In an embodiment, the user 2 selects one or more buttons on thehand-held controller 520B to provide a search query, e.g., adescription, alphanumeric characters, etc., describing or identifyingthe virtual item 502A to the search field 522. The search query data issent from the hand-held controller 520B to the computer 106 via thewired or wireless connection between the hand-held controller 520B andthe computer 106. The NIC 107 of the computer 106 packetizes the searchquery data and sends the packets via the computer network 110 to the NIC408.

The processor 406 of the server 402 receives the search query from theNIC 408, determines that the search query identifies the virtual item502A. The permission calculator 416 of the processor 406 determinesbased on the permission set 2 having permissions for the user account 2of the user 2 whether the user account 2 will be allowed to access thevirtual item 502A, the tag 508A, and the content 506 regarding thevirtual item 502A. For example, the permission set 2 does not allow theuser account 2 to access the virtual item 502A and/or the content 506and/or the tag 508A upon determining that a demographic profile storedwithin the user account 2 indicates that the user 2 is not qualified,e.g., is a minor, does not have interest, does not meet the demographicthreshold, etc.

Upon determining that the user account 2 has permission based on thepermission set 2 to access the virtual item 502A, the processor 406searches the memory device 410 for the model of the virtual item 502Abased on the identifier IDv1 of the virtual item 502A, for the content506 based on the identifier IDa1 of the content 506, and for the tag508A that is associated with the content 506. The processor 406 sendsthe model data, content data, and tag data for the virtual item 502A tothe encoder of the server 402. The encoder of the server 402 encodes themodel, tag, and content data and sends the encoded model, tag, andcontent data to the NIC 408. The NIC 408 packetizes the encoded model,tag, and content data to generate packets, and sends the packets via thecomputer network 110 to the NIC 107 of the computer 110. The NIC 107 ofthe computer 106 depacketizes the encoded model, tag, and content datato extract the encoded model, tag, and content data and provides theencoded model, tag, and content data to the decoder of the computer 106.The decoder of the computer 106 decodes the encoded model, tag, andcontent data to output the model, tag, and content data and provides themodel, tag, and content data to the display device of the HMD 404B viathe wired or the wireless connection for display of thethree-dimensional view of the virtual item 502A, the content 506, andthe tag 508A on the display device of the HMD 404B.

In one embodiment, the NIC 408 packetizes the encoded model, tag, andcontent data to generate packets, and sends the packets via the computernetwork 110 to the NIC of the HMD 404B. The NIC of the HMD 404Bdepacketizes the encoded model, tag, and content data to extract theencoded model, tag, and content data and provides the encoded model,tag, and content data to the decoder of the HMD 404B. The decoder of theHMD 404B decodes the encoded model, tag, and content data to output themodel, tag, and content data and provides the model, tag, and contentdata to the display device of the HMD 404B for display of thethree-dimensional view of the virtual item 502A, the content 506, andthe tag 508A on the display device of the HMD 404B.

In one embodiment, when the processor 406 determines that the useraccount 2 has permission to access the virtual item 502A, the ratingdeterminator 412, shown in FIG. 4A, of the processor 406 determineswhether the rating 1 assigned to the virtual item 502A is greater thanthe pre-determined rating and the ratings assigned to other virtualitems in the virtual environment A2 is less than the pre-determinedrating. Upon determining that the virtual item 502A is assigned therating 1 that is greater than the pre-determined rating and the ratingsassigned to other virtual items in the virtual environment A2 is lessthan the pre-determined rating, the processor 406 determines to displaythe virtual item 502A within the virtual environment A2 in a moreprominent location compared to other virtual items in the virtualenvironment A2. For example, the processor 406 sends with the virtualitem data via the NIC 408 and the computer network 110 to the computer106 for displaying the virtual item 502A in the virtual environment A2,the prominent location at which the virtual item 502A is to be displayedin the virtual environment A2. The computer 106 receives the dataindicating the prominent location and provides the data to the displaydevice of the HMD 404B to display the virtual item 502A in the prominentlocation compared to the virtual items 502B through 502E. For example,the virtual item 502A has less depth on the display device of the HMD404B compared to depth of the virtual items 502B through 502E on thedisplay device of the HMD 404B.

In one embodiment, the processor 406 sends with the virtual item datavia the NIC 408 and the computer network 110 to the HMD 404B fordisplaying the virtual item 502A in the virtual environment A2, theprominent location at which the virtual item 502A is to be displayed inthe virtual environment A2. The prominent location is not sent to thecomputer 106. The HMD 404B receives the data indicating the prominentlocation and provides the data to the display device of the HMD 404B todisplay the virtual item 502A in the prominent location compared to thevirtual items 502B through 502E

In various embodiments, instead of displaying the virtual item 502A inthe prominent location, the virtual item 502A is highlighted, e.g.,bolded, enlarged, displayed using a different color, displayed using adifferent shading, displayed using a different texture, or a combinationthereof, etc., compared to other virtual items 502B through 502E on thedisplay device of the HMD 404B.

FIG. 5C is a diagram of an embodiment of a virtual environment A21 toillustrate building of the additional content 550 on top of the content506. Upon viewing the content 506, the user 2 uses the controller 520Bor performs a gesture to select the tag 508A or the content 506 or thevirtual item 502A. To illustrate, the user 2 points his/her finger atthe content 506 to select the content 506. As another illustration, theuser 2 selects a button on the hand-held controller 520B operated by theuser 2 to select the content 506.

Data indicating the gesture to select the tag 508A or the content 506 orthe virtual item 502A is communicated from the HMD 404B or from thecamera 108 (FIG. 1) via the computer network 110 (FIG. 1) and the NIC408 (FIG. 4A) to the processor 406. For example, when the content 506 isselected, an indication, e.g., data indicating the selection, etc., issent from the HMD 404B via the NIC 107 of the computer 106, the computernetwork 110, and the NIC 408 to the processor 406. In one embodiment,the indication of the selection of the content 506 is packetized by theNIC of the HMD 404B and sent from the NIC of the HMD 404B via thecomputer network 110 and the NIC 408 to the processor 406. When thehand-held controller 520B is used, data indicating the selection of thetag 508A or the content 506 or the virtual item 502A is sent from thehand-held controller 520B via the computer 106 and the computer network110 (FIG. 1) and the NIC 408 (FIG. 4A) to the processor 406.

Upon receiving the data indicating the gesture or the selection, theprocessor 406 generates additional option data, e.g., data forgenerating a field for entry of text, data for enabling entry of audiocontent, data for enabling receipt of video content, a button forenabling entry of the audio content, a button for enabling recording ofthe video content, etc. The NIC 408 packetizes and sends the additionaloption data via the computer network 110 to the NIC 107 of the computer106, which sends the additional option data via the wired or wirelessconnection to the HMD 404B. In one embodiment, the additional optiondata is sent from the NIC 408 via the computer network 110 to the NIC ofthe HMD 404B without using the computer 106.

The additional option data is displayed as an option 552, e.g., an audiobutton, a video recording button, a text entry field, etc., on the HMD404B for entry of the additional content 550. For example, a text entryfield is displayed within the virtual environment A2. The user 2 makes agesture to select the text entry field and provides the additionalcontent 550, e.g., text, graphics, etc. As another example, the user 2clicks the hand-held controller 520B to select the text entry field andfurther selects various alphanumeric characters displayed on the HMD404B to provide the additional content 550. As yet another example, theuser 2 provides the additional content 550, such as sound data, into amicrophone of the HMD 404B. As still another example, the user 2provides the additional content 550, such as video content, to thecamera 108 or the camera on the HMD 404B. Examples of the additionalcontent 550 include text, audio, video, graphics, or a combinationthereof. Illustrations of the additional content 550 include “I too lovethis purse”, “I agree that this purse is awesome”, “I do not like thispurse as well”, “This purse is alright according to my friend”, “Thispurse lasted forever for me”, “I agree with user ID1”, a sad emoji, ahappy emoji, an upset emoji, a mad emoji, etc.

The additional content 550 is sent from the HMD 404B via the wired orwireless connection to the computer 106. The NIC 107 of the computer 106packetizes the additional content 550 and sends the packets via thecomputer network 110 to the NIC 408 of the server 402. In oneembodiment, the additional content 550 is packetized by and sent fromthe NIC of the HMD 404B via the computer network 110 to the NIC 408without using the computer 106. The NIC 408 receives the additionalcontent 550 and provides the additional content 550 to the processor 406of the server 402.

The processor 406 associates, e.g., establishes a one-to-onecorrespondence between, establishes a mapping between, a link between,establishes a unique relationship between, etc., the virtual item 502Aand the additional content 550, creates a link between the additionalcontent 550 and the user ID2 of the user 2, and creates a link betweenthe additional content 550 and the tag 508A. For example, with referenceto FIG. 4B, the processor 406 modifies the entry 461 in the database452. The entry 461 includes storage in the database 452 of content IDa2assigned to the additional content 550 by the processor 406. Also, theentry 461 includes storage in the database 452 of the user ID2. The userID2 is assigned to the user account 2 by the processor 406. For example,the user ID2 includes a user name, or a password, or a combinationthereof. The user ID2 is provided by the user 2 to authenticate the useraccount 2. The processor 406 establishes a link 463 between the virtualitem IDv1 and the content IDa2 and creates a link 465 between thecontent IDa2 and the user ID2. To illustrate, the processor 406 createsa pointer from a memory region, e.g., a memory cell, of the memorydevice 410 that stores the virtual item IDv1 to a memory region of thememory device 410 that stores the content IDa2. Moreover, the processor406 creates a pointer from a memory region of the memory device 410 thatstores the user ID2 to a memory region of the memory device 410 thatstores the content IDa2. The processor 406 establishes a link 463between the content IDa2 and the tag identifier TI1. As an example, theprocessor 406 creates a pointer from a memory region of the memorydevice 410 that stores the tag identifier TI1 to a memory region of thememory device 410 that stores the content IDa2. When the user 3 selectsthe tag 508A, the content 506 and the content 550 are displayed on anHMD worn by the user 3.

FIG. 5D is a diagram to illustrate different perspective views of thevirtual item 502A based on a gesture made by a real user, e.g., the user1, the user 2, etc. When the real user who is wearing an HMD has the HMDfacing a side S1 of the virtual item 502A, the virtual item 502A, thecontent 506, and/or the tag 508A are displayed on the HMD in a firstorientation, e.g., perspective, etc. However, when the user wearing theHMD moves his/her head to change an orientation of the HMD to face aside S2 of the virtual item 502A, the virtual item 502A, the content506, and/or the tag 508A are displayed on the HMD in a secondorientation, e.g., perspective, etc. The second orientation forms anangle, e.g., 90 degrees, between 70 and 90 degrees, etc., with respectto the first orientation. When the virtual item 502A is displayed in thesecond orientation, the side S2 of the virtual item 502A that was notdisplayed before in the first orientation is now displayed on the HMD.

Similarly, when the user wearing the HMD moves his/her head to change anorientation of the HMD to face a side S3 of the virtual item 502A, thevirtual item 502A, the content 506, and/or the tag 508A are displayed onthe HMD in a third orientation, e.g., perspective, etc. The thirdorientation forms an angle, e.g., 90 degrees, between 70 and 90 degrees,etc., with respect to the second orientation. As an example, the thirdorientation of the content 506 is a mirror image of the firstorientation of the content 506. As another example, in both the firstand third orientations, the content 506 and the tag 508A are displayedwithin a vertical plane. When the virtual item 502A is displayed in thethird orientation, the side S3 of the virtual item 502A that was notdisplayed before in the first orientation and in the second orientationis now displayed on the HMD.

Also, when the user wearing the HMD moves his/her head to change anorientation of the HMD to face a side S4 of the virtual item 502A, thevirtual item 502A is displayed on the HMD in a fourth orientation, e.g.,perspective, etc., with the content 506 in the fourth orientation and/orthe tag 508A in the fourth orientation. The fourth orientation forms anangle, e.g., 90 degrees, between 70 and 90 degrees, etc., with respectto the third orientation. As an example, in the fourth orientation, thecontent 506 and the tag 508A are displayed within a horizontal plane.When the virtual item 502A is displayed in the fourth orientation,another side S4 of the virtual item 502A that was not displayed beforein the first orientation, the second orientation, and the thirdorientation is now displayed on the HMD. As illustrate, the side S1 isadjacent to the side S2. The side S3 is adjacent to the side S2 and isseparated from the side S1 by the side S2. Moreover, the side S4 isadjacent to the side S2, the side S1, and the side S3.

With reference to FIG. 4B, the database 452 stores a model of thevirtual item 502A to facilitate the different perspective views of thevirtual item 502A. The different perspectives are displayed within avirtual environment, such as the virtual environment A1 or the virtualenvironment A2. For example, the database 452 stores a series of imageframes, which are combined by the processor 406 to generate athree-dimensional model of the virtual item 502A. Each of the imageframes includes graphics data, e.g., data of a geometry of the virtualitem 502A, data of a shape of the virtual item 502A, data of shading ofthe virtual item 502A, data of one or more colors the virtual item 502A,data of one or more light intensities of the virtual item 502A, data ofa texture of the virtual item 502A, or a combination thereof, etc. As anexample, the texture is a function of lighting of the virtual item 502A,a number of polygonal pieces that are combined and arranged to form thevirtual item 502A, and one or more colors of the virtual item 502A.

The model is encoded by the encoder of the server 402, is thenpacketized and sent from the NIC 408 via the computer network 110 to theNIC 107 of the computer 106. The NIC 107 of the computer 106depacketizes the model to provide encoded modeled data to the decoder ofthe computer 106. The decoder of the computer 106 decodes the encodedmodeled data to output the modeled data, which is sent from the computer106 via the wired or wireless connection to the HMD. The model is thendisplayed on the display device of the HMD.

In one embodiment, the model is encoded by the encoder of the server402, is then packetized and sent from the NIC 408 via the computernetwork 110 to the NIC of the HMD. The NIC of the HMD depacketizes themodel to provide encoded modeled data to the decoder of the HMD. Thedecoder of the HMD decodes the encoded modeled data to output themodeled data and the model is displayed on the display device of theHMD.

The camera 108 captures the different movements of the head of the userto generate image data. The camera 108 sends the image data to the NIC107 of the computer 110 via the wired or wireless connection. The NIC107 of the computer 110 packetizes the image data and sends the imagedata via the computer network 110 to the NIC 408.

The processor 406 receives the image data from the NIC 408 and executesthe computer program to change a perspective view of the virtual item502A, a perspective view of the content 506, and/or a perspective viewof the tag 508A based on a position and orientation of the head of theuser in the image data. The processor 406 sends the perspective viewdata for the position and orientation of the head to the encoder of theserver 402. The encoder encodes the perspective view data and sends theencoded data to the NIC 408, which packetizes the encoded data. The NIC408 sends the packets having the encoded data via the computer network110 to the NIC 107 of the computer 110. The NIC 107 of the computer 106depacketizes the encoded data and the decoder of the computer 106decodes the encoded data to output the perspective view data. Theperspective view data is sent via the wired or wireless connection fromthe computer 106 to the HMD for display on the display device of theHMD.

In one embodiment, the NIC 408 sends the packets having the encoded datavia the computer network 110 to the NIC of the HMD. The NIC of the HMDdepacketizes the encoded data and the decoder of the HMD decodes theencoded data to output the perspective view data for display on thedisplay device of the HMD.

In one embodiment, instead of the head movement, another body part,e.g., a finger, one or more fingers, etc., of the user is moved andcaptured by the camera 108 or by the camera of the HMD to change theperspective of the virtual item 502A.

FIG. 6A is a diagram of an embodiment of a virtual environment A3 toillustrate a placement of the virtual item 502A in a location that ismore prominent than locations of the virtual items 502B and 502C withina virtual environment A3. The processor 406 determines whether therating 1 of the virtual item 502A exceeds a predetermined rating. Forexample, when a number of comments made by one or more users via his/heruser accounts on the virtual item 502A exceeds a number of comments madeby one or more users via his/her user account on another virtual item,e.g., the virtual item 502B, or 502C, or 502D, or 502E, by a pre-setamount, the rating 1 of the virtual item 502A exceeds the pre-determinedrating and the ratings of the other virtual item is below thepre-determined rating. As another example, when a number of positivecomments made by one or more users via his/her user accounts on thevirtual item 502A exceeds a number of positive comments made by one ormore users via his/her user account on another virtual item by a pre-setlimit, the rating 1 of the virtual item 502A exceeds the pre-determinedrating and the ratings of the other virtual item is below thepre-determined rating. An example of a positive comment on a virtualitem is a comment that further promotes a virtual item or that praisesthe virtual item or that provides a high grade to the virtual item or acombination thereof. As another example, when a number of negativecomments made by one or more users via his/her user accounts on thevirtual item 502A is less than a number of negative comments made by oneor more users via his/her user account on another virtual item by apre-set number, the rating 1 of the virtual item 502A exceeds thepre-determined rating and the ratings of the other virtual item is belowthe pre-determined rating. An example of a negative comment on a virtualitem is a comment that demotes a virtual item or that degrades thevirtual item or that provides a low grade to the virtual item or acombination thereof. It should be noted that a comment made on a virtualitem includes data regarding availability of the virtual item, or aquality of the virtual item, or a look of the virtual item, or a designof the virtual item, or a combination thereof.

Moreover, the processor 406 determines whether the user 3 logs intohis/her user account 3 to access the virtual store represented by thevirtual environments A1 and A2. For example, the user 3 makes a gestureor selects, via a hand-held controller 520C, an identifier of thevirtual store displayed on the HMD 404C to indicate to the processor 406that the user 3 desires to access the virtual store. Upon determiningthat the user 3 desires to access the virtual store, the processor 406sends, via the computer network 110, virtual environment data to displaythe virtual environment A3 for display of the virtual item 502A on theHMD 404C in a more prominent manner, e.g., in front of the virtualstore, in a larger format, in a larger size, in a highlighted format, ina bolded format, etc., compared to a manner in which other virtual items502B and 502C are displayed within the virtual environment A3. In thismanner, the virtual item 502A is pushed to the user account of the user3 by being displayed in the more prominent location.

In one embodiment, a virtual item is presented in the prominent mannerto a user who satisfies the demographic threshold in addition to meetingor exceeding the predetermined rating.

FIG. 6B is a diagram of an embodiment of a virtual environment A4 toillustrate restriction from access to the virtual items 502D and 502E bythe user 4 whose user account 4 is restricted. The user 4 logs intohis/her user account 4 and makes a selection, either via a gesture or ahand-held controller 520D, of the identifier of the virtual store toaccess the virtual environment A4 of the virtual store. An indication ofthe selection is sent from the HMD 404D via the computer 106, thecomputer network 110, and the NIC 408 to the processor 406. In oneembodiment, the indication of the selection is sent from the HMD 404Dvia the computer network 110 and the NIC 408 to the processor 406without using the computer 106.

Upon determining that the user 4 desires to access the virtual store,the processor 406 determines based on the permission set 4 whether theuser account 4 is restricted from accessing the virtual items 502D and502E. Upon determining that the user account 4 is restricted, theprocessor 406 sends via the encoder of the server 402, the NIC 408, andthe computer network 110, virtual environment data that excludes virtualitem data for display of the virtual environment A4 on the HMD 404D aslacking the virtual items 502D and 502E. For example, the encoder of theserver 402 encodes the virtual environment data excluding the virtualitem data to generate encoded virtual environment data and sends theencoded virtual environment data to the NIC 408. The NIC 408 packetizesthe encoded virtual environment data and sends the packets via thecomputer network 110 to the computer 106. The NIC 107 of the computer106 depacketizes the packets to extract the encoded virtual environmentdata and the decoder of the computer 106 decodes the encoded virtualenvironment data to output the virtual environment data. The virtualenvironment data is sent from the computer 106 to the HMD 404D via awired or wireless connection to display the virtual environment A4 onthe display device of the HMD 404D. The virtual environment A4 excludesthe virtual items 502D and 502E.

In one embodiment, the virtual environment data excluding the virtualitems 502D and 502E is sent to the HMD 404D. A NIC of the HMD 404Ddepacketizes the packets to extract the encoded virtual environment dataand a decoder of the HMD 404D decodes the encoded virtual environmentdata to output the virtual environment data to display the virtualenvironment A4 on the display device of the HMD 404D.

In various embodiments, instead of excluding the virtual item data forthe virtual items 502D and 502E, the processor 406 determines to hide orcover the virtual items 502D and 502E within the virtual environment A4.

It should be noted that the virtual environments A1 through A4 representthe same virtual store, e.g., store owned by the same entity, e.g.,online retailer, etc., or that has the same lessee, etc. For example,each of the virtual environments A1 through A4 is a view of the virtualstore from different standpoints, e.g., left, right, back, front, etc.In various embodiments, instead of the different standpoints, the samestandpoint is used to represent the virtual store on the HMDs 404Athrough 404D.

FIG. 7A is a diagram of a virtual environment B1 that is displayed onthe HMD 404A. The virtual environment B1 has a virtual item 702, e.g., avirtual swimming pool, etc. of a facility, e.g., a hotel, a lodge, amotel, etc. The virtual item 702 is tagged in a manner similar to whichthe virtual item 502A (FIG. 5A) is tagged. For example, the virtual item702 is tagged with a tag 706 when the virtual item 702 is associatedwith content 704, such as, a comment, e.g., “the pool is gorgeous”,“watch out, the pool is dangerous”, etc., which is positive or negative.The content 704 associated with the virtual item 702 is provided by theuser 1.

FIG. 7B is a diagram of an embodiment of a virtual environment B2 thatis displayed on the HMD 404B. When the user 2 selects the tag 706 or thevirtual item 702, the content 704 associated with the tag 706 or thevirtual item 702 is displayed on the HMD 404B.

FIG. 8 is a diagram of an embodiment of a virtual environment B3 that isdisplayed on the HMD 404C to illustrate a display of the virtual item702 in the prominent manner. For example, during a virtual tour of thefacility, the processor 406 displays the virtual item 702 first comparedto other virtual locations of the facility. For example, a pool area ofthe facility is displayed first compared to a lobby of the hotel and aroom of the hotel. When the user 3 makes a gesture or uses the hand-heldcontroller 520C to access a website of the hotel from the processor 406via the computer network 110 and the NIC 408, the website includes thepool area. The website includes buttons that the user 3 selects to visitother areas, such as the lobby and the room of the hotel. As anotherillustration, when the user 3 makes a gesture or uses the hand-heldcontroller 520C to access a website of the hotel from the processor 406via the computer network 110 and the NIC 408, the website runs a slideshow to display the pool area first before displaying the lobby and thehotel. The virtual item 702 has a higher rating compared to othervirtual items of the facility.

It should be noted that the virtual environments B1, B2, and B3 areviews from the same standpoint of the virtual item 702.

In some embodiments, a virtual item is an item displayed using virtualreality. For example, the virtual item 702 is a computer-generatedrepresentation of a real pool.

In one embodiment, educational content, e.g., videos, videos of how tofix cars, videos taken during a classroom lecture, etc., is tagged byusing the methods described herein. For example, an augmented realityscene is displayed on HMDs of students. Various items, e.g., a DNAsequence, a mathematical formula, a chemical equation, etc., drawn on areal classroom board in a real classroom environment are tagged whencontent is associated with the various items. The content is receivedfrom the students sitting in the classroom. The students are examples ofthe users 1 through 4.

As another example, various parts of a car engine are tagged toassociate the parts with content that is provided by two mechanics, suchas the users 1 and 2. The two mechanics wear HMDs to provide content,e.g., “this is how you change oil”, “this is an air filter”, etc., totag various parts of the car and a real-time video of the various partsis displayed on the HMDs.

FIG. 9 is a diagram of an embodiment of a virtual environment C that isdisplayed on the HMD 404B. The virtual environment C is displayed on theHMD 404B when a game code is executed by the processor 406 to allow theuser 2 to play a game. The virtual environment C displays variousvirtual items, e.g., a virtual boat 902A, a virtual cave 902B, etc.,that are tagged in a manner similar to which the virtual item 502A (FIG.5A) is associated with a tag. The user 2 navigates the virtualenvironment C during the play of the game and selects a tag 904A via thehand-held controller 520B and/or by making a gesture. The tag 904A isassociated with the virtual item 902A. Upon receiving an indication ofthe selection of the tag 904A, the processor 406 (FIG. 4A) sends contentdata to display content, e.g., clue, etc., associated with the tag 904Aand the virtual item 902A. The content associated with the tag 904Aprovides a clue to the user 2 to unlock another level in the game orprovides a clue to the the user 2 to gather more points in the game or acombination thereof. Similarly, a tag 904B when selected by the user 2allows the user 2 to view content associated with the virtual item 902B.The content associated with the virtual item 902B and the tag 904Bprovides a clue to the user 2 to unlock another level in the game orprovides a clue to the user 2 to gather more points in the game or acombination thereof.

It should be noted that content associated with a tag displayed withinthe game is created by another user, such as the user 1, while the user1 is playing the game. For example, the user 1 makes a gesture or usesthe hand-held controller 520A to access the game from the processor 406via the computer network 110 and the NIC 408. The user 1 provides aninput to the hand-held controller 520A or makes a gesture to provide thecontent associated with the virtual item 902A to create the tag 904A.

In one embodiment, one or more of the virtual environments describedherein includes a search field, which is selected by a user byperforming a gesture or by selecting a button on a hand-held controller,to search for content associated with tags. For example, during the playof the game, a user performs a gesture and/or uses the hand-heldcontroller and/or provides an audio input to provide one or morealphanumeric characters in the search field. Upon receiving thealphanumeric characters from an HMD via the computer network 110 or viathe computer 106 and the computer network 110, the processor 406searches for content that matches a portion of or all of thealphanumeric characters and provides tags and/or the content for displayon an HMD worn by the user. In an embodiment, once a user is at a gamestate, e.g., a scene, a point level, etc. in the game, the user searchesfor tags.

In an embodiment, the tags and/or content associated with the tagsappears based on a gesture performed by a user. For example, when theuser stares at a particular virtual item in a virtual scene, theprocessor 406 determines that the user is staring, e.g., looking at thevirtual item for greater than a pre-determined amount of time, etc., oris making a gesture to move closer to the virtual item, and determinesto display content and/or tags associated with the virtual item within auser account of the user on an HMD worn by the user. To furtherillustrate, the gaze tracking camera 312 (FIG. 3) detects the gaze ofthe user 1 at the virtual item. The gaze tracking camera 312 capturesimages of the gaze and communicates the images as motion input 318 viathe computer network 110 and the NIC 408 (FIG. 4A) or via the computer106 and the computer network 110 and the NIC 408 to the game engine 320,which is being executed by the processor 406. The processor 406determines whether the images indicate that the user is gazing at thevirtual item. Upon determining that the user is gazing at the virtualitem, the processor 406 determines to display content and/or tagsassociated with the virtual item.

In one embodiment, the processor 406 generates a slider that isdisplayed on an HMD. A user wearing the HMD makes a gesture and/or usesa hand-held controller to change a position of a bar on the slider toprovide an indication to the processor 406 of a number of tags to bepresented in a virtual environment. Various positions of the bar on theslider indicate whether to present tags associated with content receivedfrom social network friends of the user via a social network, or fromeveryone, or no one, or people of the same demographic as that of theuser, etc. In various embodiments, the processor 406 provides options onthe HMD to present tags associated with content received from socialnetwork friends, or from everyone, or no one, or people of the samedemographic as that of the user, etc.

With reference to FIG. 10, a diagram illustrating components of the HMD102 is shown, in accordance with an embodiment of the disclosure. TheHMD 102 includes a processor 1000 for executing program instructions. Amemory device 1002 is provided for storage purposes, and may includeboth volatile and non-volatile memory. A display device 1004 is includedwhich provides a visual interface that a user may view. A battery 1006is provided as a power source for the HMD 102. A motion detection module1008 includes any of various kinds of motion sensitive hardware, such asa magnetometer 1010, an accelerometer 1012, and a gyroscope 1014.

An accelerometer is a device for measuring acceleration and gravityinduced reaction forces. Single and multiple axis models are availableto detect magnitude and direction of the acceleration in differentdirections. The accelerometer is used to sense inclination, vibration,and shock. In one embodiment, three accelerometers 1012 are used toprovide the direction of gravity, which gives an absolute reference fortwo angles (world-space pitch and world-space roll).

A magnetometer measures the strength and direction of the magnetic fieldin the vicinity of the HMD 102. In one embodiment, three magnetometers1010 are used within the HMD 102, ensuring an absolute reference for theworld-space yaw angle. In one embodiment, the magnetometer is designedto span the earth magnetic field, which is ±80 microtesla. Magnetometersare affected by metal, and provide a yaw measurement that is monotonicwith actual yaw. The magnetic field is warped due to metal in theenvironment, which causes a warp in the yaw measurement. If necessary,this warp is calibrated using information from other sensors such as thegyroscope or the camera. In one embodiment, accelerometer 1012 is usedtogether with magnetometer 1010 to obtain the inclination and azimuth ofthe HMD 102.

In some implementations, the magnetometers of the HMD 102 are readduring times when electromagnets in other nearby devices are inactive.

A gyroscope is a device for measuring or maintaining orientation, basedon the principles of angular momentum. In one embodiment, threegyroscopes 1014 provide information about movement across the respectiveaxis (x, y and z) based on inertial sensing. The gyroscopes help indetecting fast rotations. However, the gyroscopes can drift overtimewithout the existence of an absolute reference. The gyroscopes are resetperiodically, and the resetting done using other available information,such as positional/orientation determination based on visual tracking ofan object, accelerometer, magnetometer, etc.

A camera 1016 is provided for capturing images and image streams of thereal-world interactive environment. More than one camera is included inthe HMD 102, including a camera that is rear-facing (directed away froma user when the user is viewing the display of the HMD 102), and acamera that is front-facing (directed towards the user when the user isviewing the display of the HMD 102). Additionally, in some embodiments,a depth camera 1018 is included in the HMD 102 for sensing depthinformation of objects in a real environment.

The HMD 102 includes speakers 1020 for providing audio output. Also, insome embodiments, a microphone 1022 is included for capturing audio fromthe real environment, including sounds from the ambient environment,speech made by the user, etc. The HMD 102 includes tactile feedbackmodule 1024 for providing tactile feedback to the user. In oneembodiment, the tactile feedback module 1024 is capable of causingmovement and/or vibration of the HMD 102 so as to provide tactilefeedback to the user.

LEDs 1026 are provided as visual indicators of statuses of the HMD 102.For example, an LED may indicate battery level, power on, etc. A cardreader 1028 is provided to enable the HMD 102 to read and writeinformation to and from a memory card. A USB interface 1030 is includedas one example of an interface for enabling connection of peripheraldevices, or connection to other devices, such as other portable devices,computers, etc. In various embodiments of the HMD 102, any of variouskinds of interfaces may be included to enable greater connectivity ofthe HMD 102.

A WiFi module 1032 is included for enabling connection to the Internetor a local area network via wireless networking technologies. Also, theHMD 102 includes a Bluetooth module 1034 for enabling wirelessconnection to other devices. In some embodiments, a communications link1036 is included for connection to other devices. In one embodiment, thecommunications link 1036 utilizes infrared transmission for wirelesscommunication. In other embodiments, the communications link 1036utilizes any of various wireless or wired transmission protocols forcommunication with other devices.

Input buttons/sensors 1038 are included to provide an input interfacefor the user. Any of various kinds of input interfaces are included,such as buttons, touchpad, joystick, trackball, etc. An ultra-soniccommunication module 1040 may be included in HMD 102 for facilitatingcommunication with other devices via ultra-sonic technologies.

Bio-sensors 1042 are included to enable detection of physiological datafrom a user. In one embodiment, the bio-sensors 1042 include one or moredry electrodes for detecting bio-electric signals of the user throughthe user's skin.

A video input 1044 receives a video signal from a primary processingcomputer (e.g. main game console) for rendering on the HMD 102. In someimplementations, the video input is a high-definition multimedia input(HDMI) input.

The foregoing components of HMD 102 have been described as merelyexemplary components that are included in HMD 102. In variousembodiments of the disclosure, the HMD 102 includes or does not includesome of the various aforementioned components. Embodiments of the HMD102 additionally includes other components not presently described, butknown in the art, for purposes of facilitating aspects of the presentdisclosure as herein described.

FIG. 11 is a block diagram of a game system 1100, according to variousembodiments of the disclosure. The game system 1100 provides a videostream to one or more clients 1110 via the computer network 110. Thegame system 1100 typically includes a video server system 1120 and anoptional game server 1125. The server 402 (FIG. 4A) is a part of theserver system 1120. In some embodiments, the game server 1125 is locatedwithin the video server system 1120 and the server 402 is a part of thevideo server system 1120. The video server system 1120 provides thevideo stream to the one or more clients 1110, e.g., HMDs describedherein, the computer 106 (FIG. 1), etc., with a minimal quality ofservice. For example, the video server system 1120 receives a gamecommand that changes the state of or a point of view within a videogame, and provides the clients 1110 with an updated video streamreflecting this change in state with minimal lag time. The video serversystem 1120 provides the video stream in a wide variety of alternativevideo formats, including formats yet to be defined. Further, as anexample, the video stream includes video frames for presentation to auser at a wide variety of frame rates. Typical frame rates are 30 framesper second, 60 frames per second, and 120 frames per second. Althoughhigher or lower frame rates are included in alternative embodiments ofthe disclosure.

The clients 1110, referred to herein individually as 1110A, 1110B, etc.,include head mounted displays, terminals, personal computers, gameconsoles, tablet computers, telephones, set top boxes, kiosks, wirelessdevices, digital pads, stand-alone devices, handheld game playingdevices, and/or the like. Typically, the clients 1110 receive encodedvideo streams, decode the video streams, and present the resulting videoto a user, e.g., a player of a game. The processes of receiving encodedvideo streams and/or decoding the video streams typically includesstoring individual video frames in a receive buffer of the client 1110Aor 1110B. The video streams are presented to a user on a displayintegral to the client 1110A or 1110B or on a separate device such as amonitor or television. The clients 1110 optionally support more than onegame player. For example, a game console is configured to support two,three, four or more simultaneous players. Each of these players receivea separate video stream, or a single video stream includes regions of aframe generated specifically for each player, e.g., generated based oneach player's point of view. The clients 1110 are optionallygeographically dispersed. The number of clients 1110 included in gamesystem 1100 varies widely from one or two to thousands, tens ofthousands, or more. As used herein, the term “game player” is used torefer to a person that plays a game and the term “game playing device”is used to refer to a device used to play a game. In some embodiments,the game playing device refers to a plurality of computing devices thatcooperate to deliver a game experience to the user. For example, a gameconsole and an HMD cooperate with the video server system 1120 todeliver a game viewed through the HMD. In one embodiment, the gameconsole receives the video stream from the video server system 1120, andthe game console forwards the video stream, or updates to the videostream, to the HMD for rendering.

The clients 1110 receive video streams via the computer network 110. Thecomputer network 110 is any type of communication network including theInternet, wireless networks, local area networks, wide area networks,private networks, and/or the like. In typical embodiments, the videostreams are communicated via standard protocols, such as TCP/IP or UserDatagram Protocol over IP (UDP/IP). Alternatively, the video streams arecommunicated via proprietary standards.

A typical example of clients 1110 is a personal computer including aprocessor, non-volatile memory, a display, decoding logic, networkcommunication capabilities, and input devices. The decoding logicincludes hardware, firmware, and/or software stored on a computerreadable medium. Systems for decoding (and encoding) video streams varydepending on the particular encoding scheme used.

In some embodiments, the clients 1110 further include systems formodifying received video. For example, a client performs furtherrendering, to overlay one video image on another video image, to crop avideo image, and/or the like. For example, clients 1110 receive varioustypes of video frames, such as I-frames, P-frames and B-frames, and toprocess these frames into images for display to a user. In someembodiments, clients 1110 perform further rendering, shading, conversionto three-dimensional (3-D) images, or like operations on the videostream. The clients 1110 optionally receive more than one audio or videostream. Input devices of the clients 1110 include, for example, aone-hand game controller, a two-hand game controller, a gesturerecognition system, a gaze recognition system, a voice recognitionsystem, a keyboard, a joystick, a pointing device, a force feedbackdevice, a motion and/or location sensing device, a mouse, a touchscreen, a neural interface, a camera, input devices yet to be developed,and/or the like.

The video stream (and optionally audio stream) received by the clients1110 is generated and provided by the video server system 1120. Thisvideo stream includes video frames (and the audio stream includes audioframes). The video frames contribute (e.g., they include pixelinformation in an appropriate data structure) meaningfully to the imagesdisplayed to the user. As used herein, the term “video frames” is usedto refer to frames including predominantly information that contributesto, e.g. to effect, the images shown to the user. Most of the teachingsherein with regard to “video frames” are also applied to “audio frames”,in some embodiments.

The clients 1110 receive inputs from a user via one or more of the inputdevices. These inputs include game commands that change the state of thevideo game or otherwise affect game play. The game commands are receivedusing the input devices and/or are automatically generated by computinginstructions executing on clients 1110. The received game commands arecommunicated from the clients 1110 via the computer network 110 to thevideo server system 1120 and/or the game server 1125. For example, insome embodiments, the game commands are communicated to the game server1125 via the video server system 1120. In some embodiments, separatecopies of the game commands are communicated from the clients 1110 tothe game server 1125 and the video server system 1120. The communicationof game commands is optionally dependent on the identity of the command.Game commands are optionally communicated from the client 1110A througha different route or communication channel that that used to provideaudio or video streams to the client 1110A.

The game server 1125 is optionally operated by a different entity thanthe video server system 1120. For example, the game server 1125 isoperated by a publisher of a multiplayer game. In this example, thevideo server system 1120 is optionally viewed as a client by the gameserver 1125 and optionally configured to appear from the point of viewof the game server 1125 to be a client executing a game engine.Communication between the video server system 1120 and the game server1125 optionally occurs via the computer network 110. As such, the gameserver 1125 is a multiplayer game server that sends game stateinformation to multiple clients, one of which is the video server system1120. The video server system 1120 communicates with multiple instancesof the game server 1125 at the same time. For example, the video serversystem 1120 provides a plurality of different video games to differentusers. Each of these different video games are supported by a differentgame server and/or published by different entities. In some embodiments,several geographically distributed instances of the video server system1120 provide game video to a plurality of different users. Each of theseinstances of the video server system 1120 is in communication with thesame instance of the game server 1125. Communication between the videoserver system 1120 and one or more game servers optionally occurs via adedicated communication channel. For example, the video server system1120 is connected to the game server 1125 via a high bandwidth channelthat is dedicated to communication between these two systems.

The video server system 1120 includes at least a video source 1130, aninput/output (I/O) device 1145, a processor 1150, and non-transitorystorage 1155. The video server system 1120 includes one computing deviceor be distributed among a plurality of computing devices. Thesecomputing devices are optionally connected via a communications systemsuch as a local area network.

The video source 1130 provides a video stream, e.g., streaming video ora series of video frames that form a moving picture. In someembodiments, the video source 1130 includes a video game engine andrendering logic. The video game engine receives game commands from aplayer and to maintain a copy of the state of the video game based onthe received commands. This game state includes the position of objectsin a game environment, as well as typically a point of view. The gamestate includes, in some embodiments, properties, images, colors and/ortextures of objects. The game state is typically maintained based ongame rules, as well as game commands such as move, turn, attack, setfocus to, interact, use, and/or the like. Part of the game engine isoptionally disposed within the game server 1125. The game server 1125maintains, in various embodiments, a copy of the state of the game basedon game commands received from multiple players using geographicallydisperse clients. In these cases, the game state is provided by the gameserver 1125 to the video source 1130, where a copy of the game state isstored and rendering is performed. The game server 1125 receives gamecommands directly from the clients 1110 via the computer network 110.

The video source 1130 includes rendering logic, e.g., hardware,firmware, and/or software stored on a computer readable medium such as astorage 1155. This rendering logic creates video frames of the videostream based on the game state. All or part of the rendering logic isoptionally disposed within a graphics processing unit (GPU). Renderinglogic typically includes processing stages configured for determiningthe three-dimensional spatial relationships between objects and/or forapplying appropriate textures, etc., based on the game state andviewpoint. The rendering logic produces raw video that is then usuallyencoded prior to communication to the clients 1110. For example, the rawvideo is encoded according to an Adobe Flash® standard, .wav, H.264,H.263, On2, VP6, VC-1, WMA, Huffyuv, Lagarith, MPG-x. Xvid. FFmpeg,x264, VP6-8, realvideo, mp3, or the like. The encoding process producesa video stream that is optionally packaged for delivery to a decoder ona remote device. The video stream is characterized by a frame size and aframe rate. Typical frame sizes include 800×600, 1280×720 (e.g., 720p),1024×768, although any other frame sizes may be used. The frame rate isthe number of video frames per second. A video stream includes, in someembodiments, different types of video frames. For example, the H.264standard includes a “P” frame and a “I” frame. I-frames includeinformation to refresh all macro blocks/pixels on a display device,while P-frames include information to refresh a subset thereof. P-framesare typically smaller in data size than are I-frames. As used herein theterm “frame size” is meant to refer to a number of pixels within aframe. The term “frame data size” is used to refer to a number of bytesrequired to store the frame.

In an embodiment, the video source 1130 includes a video recordingdevice such as a camera. This camera is used to generate delayed or livevideo that can be included in the video stream of a computer game. Theresulting video stream, optionally includes both rendered images andimages recorded using a still or video camera. The video source 1130also includes, in some embodiments, storage devices configured to storepreviously recorded video to be included in a video stream. The videosource 1130 includes, in various embodiments, motion or positioningsensing devices configured to detect motion or position of an object,e.g., person, and logic configured to determine a game state or producevideo-based on the detected motion and/or position.

The video source 1130 optionally provides overlays to be placed on othervideo. For example, these overlays include a command interface, log ininstructions, messages to a game player, images of other game players,video feeds of other game players (e.g., webcam video). In embodimentsof client 1110A including a touch screen interface or a gaze detectioninterface, the overlay may include a virtual keyboard, joystick, touchpad, and/or the like. In one example of an overlay a player's voice isoverlaid on an audio stream. The video source 1130 optionally furtherincludes one or more audio sources.

In embodiments in which the video server system 1120 maintains the gamestate based on input from more than one player, each player have adifferent point of view including a position and direction of view. Thevideo source 1130 optionally provides a separate video stream for eachplayer based on their point of view. Further, the video source 1130optionally provides a different frame size, frame data size, and/orencoding to each of the clients 1110. The video source 1130 optionallyprovides 3-D video.

I/O Device 1145 for the video server system 1120 sends and/or receivesinformation such as video, commands, requests for information, a gamestate, gaze information, device motion, device location, user motion,client identities, player identities, game commands, securityinformation, audio, and/or the like. I/O Device 1145 includescommunication hardware such as a network card or modem. I/O Device 1145communicates with the game server 1125, the computer network 110, and/orthe clients 1110.

The processor 1150 executes logic, e.g. software, included within thevarious components of the video server system 1120 discussed herein. Forexample, processor 1150 is programmed with software instructions inorder to perform the functions of the video source 1130, game server1125, and/or a client qualifier 1160. The video server system 1120optionally includes more than one instance of the processor 1150. Theprocessor 1150 may also be programmed with software instructions inorder to execute commands received by the video server system 1120, orto coordinate the operation of the various elements of the game system1100 discussed herein. The processor 1150 includes one or more hardwaredevices. The processor 1150 is an electronic processor.

Storage 1155 includes non-transitory analog and/or digital storagedevices. For example, storage 1155 includes an analog storage deviceconfigured to store video frames. Storage 1155 may include a computerreadable digital storage, e.g. a hard drive, an optical drive, or solidstate storage. Storage 1115 stores (e.g. by way of an appropriate datastructure or file system) video frames, artificial frames, a videostream including both video frames and artificial frames, audio frame,an audio stream, and/or the like. Storage 1155 is optionally distributedamong a plurality of devices. In some embodiments, storage 1155 storesthe software components of the video source 1130 discussed elsewhereherein. These components are stored in a format ready to be provisionedwhen needed.

The video server system 1120 optionally further includes a clientqualifier 1160. Client qualifier 1160 remotely determines thecapabilities of a client, such as client 1110A or 1110B. Thesecapabilities include both the capabilities of the client 1110A itself aswell as the capabilities of one or more communication channels betweenthe client 1110A and the video server system 1120. For example, theclient qualifier 1160 tests a communication channel through the computernetwork 110.

The client qualifier 1160 determines (e.g., discover) the capabilitiesof the client 1110A manually or automatically. Manual determinationincludes communicating with a user of the client 1110A and asking theuser to provide capabilities. For example, in some embodiments, theclient qualifier 1160 displays images, text, and/or the like within abrowser of the client 1110A. In one embodiment, the client 1110A is anHMD that includes a browser. In another embodiment, client 1110A is agame console having a browser, which may be displayed on the HMD. Thedisplayed objects request that the user enter information such asoperating system, processor, video decoder type, type of networkconnection, display resolution, etc. of the client 1110A. Theinformation entered by the user is communicated back to the clientqualifier 1160.

Automatic determination occurs, for example, by execution of an agent onthe client 1110A and/or by sending test video to the client 1110A. Theagent includes computing instructions, such as java script, embedded ina web page or installed as an add-on. The agent is optionally providedby the client qualifier 1160. In various embodiments, the agent findsout processing power of the client 1110A, decoding and displaycapabilities of the client 1110A, lag time reliability and bandwidth ofcommunication channels between the client 1110A and the video serversystem 1120, a display type of the client 1110A, firewalls present onthe client 1110A, hardware of the client 1110A, software executing onthe client 1110A, registry entries within the client 1110A, and/or thelike.

The client qualifier 1160 includes hardware, firmware, and/or softwarestored on a computer readable medium. The client qualifier 1160 isoptionally disposed on a computing device separate from one or moreother elements of the video server system 1120. For example, in someembodiments, the client qualifier 1160 determines the characteristics ofcommunication channels between the clients 1110 and more than oneinstance of the video server system 1120. In these embodiments theinformation discovered by Client Qualifier is used to determine whichinstance of the video server system 1120 is best suited for delivery ofstreaming video to one of the clients 1110.

In some embodiments, a classifier of real world objects is used todetect and identify objects from the above mentioned cameras. Theseclassifiers, allow for the fast identification of real world objects bycontaining or being connected via internet to a database of computerrecognizable objects. For example, the classifier is able to identifyindividual objects such as a coffee table or lamp situated within thevirtual reality interactive space. As a further example, the classifieris able to identify a floor and a wall and the points at which the twomeet.

In various embodiments, an HMD includes cameras on a front face of theHMD and on a band that is worn on a back side of a user's head. Thecameras on the back side face an real-world environment in which theuser performs a gesture to protect the user from accidentally hitting anobject behind the user.

Embodiments described in the present disclosure may be practiced withvarious computer system configurations including hand-held devices,microprocessor systems, microprocessor-based or programmable consumerelectronics, minicomputers, mainframe computers and the like. Theembodiments described in the present disclosure can also be practiced indistributed computing environments where tasks are performed by remoteprocessing devices that are linked through a wire-based or wirelessnetwork.

With the above embodiments in mind, it should be understood that theembodiments described in the present disclosure can employ variouscomputer-implemented operations involving data stored in computersystems. These operations are those requiring physical manipulation ofphysical quantities. Any of the operations described herein that formpart of the embodiments described in the present disclosure are usefulmachine operations. Some embodiments described in the present disclosurealso relate to a device or an apparatus for performing these operations.The apparatus can be specially constructed for the required purpose, orthe apparatus can be a general-purpose computer selectively activated orconfigured by a computer program stored in the computer. In particular,various general-purpose machines can be used with computer programswritten in accordance with the teachings herein, or it may be moreconvenient to construct a more specialized apparatus to perform therequired operations.

Some embodiments described in the present disclosure can also beembodied as computer readable code on a computer readable medium. Thecomputer readable medium is any data storage device that can store data,which can be thereafter be read by a computer system. Examples of thecomputer readable medium include a hard drive, a NAS, a ROM, a RAM, aCD-ROM, a CD-R, a CD-RW, a magnetic tape, an optical data storagedevice, a non-optical data storage device, etc. The computer readablemedium can include computer readable tangible medium distributed over anetwork-coupled computer system so that the computer readable code isstored and executed in a distributed fashion.

It should be noted that in some embodiments, any of the embodimentsdescribed herein are combined with any of the remaining embodiments.

Moreover, although some of the above-described embodiments are describedwith respect to a gaming environment, in some embodiments, instead of agame, other environments, e.g., a video conferencing environment, etc.,is used.

Although the method operations were described in a specific order, itshould be understood that other housekeeping operations may be performedin between operations, or operations may be adjusted so that they occurat slightly different times, or may be distributed in a system whichallows the occurrence of the processing operations at various intervalsassociated with the processing, as long as the processing of the overlayoperations are performed in the desired way.

Although the foregoing embodiments described in the present disclosurehave been described in some detail for purposes of clarity ofunderstanding, it will be apparent that certain changes andmodifications can be practiced within the scope of the appended claims.Accordingly, the present embodiments are to be considered asillustrative and not restrictive, and the embodiments are not to belimited to the details given herein, but may be modified within thescope and equivalents of the appended claims.

The invention claimed is:
 1. A method comprising: sending, via acomputer network, virtual environment data for display of a sharedvirtual environment on a first head-mounted display, wherein the firsthead-mounted display is configured to be worn by a first user, whereinthe shared virtual environment includes a three-dimensional view of avirtual item; receiving, via the computer network and a first useraccount of the first user, an indication of a selection associated withthe virtual item; sending, via the computer network, option data forallowing entry of content regarding the virtual item upon receiving theindication of the selection; receiving, via the computer network, thecontent; associating the content with the virtual item and the firstuser account; sending, via the computer network, tagged data fordisplaying a tag associated with the virtual item, wherein the tagindicates that the content is associated with the virtual item;analyzing a number of interactions with the virtual item including theselection of the virtual item to determine a rating for the virtualitem, wherein the interactions include three-dimensional movement of thevirtual item in the shared virtual environment; sending, via thecomputer network, data for display of the virtual item within the sharedvirtual environment to a second head-mounted display based on the ratingand an interest profile associated with a second user account of asecond user.
 2. The method of claim 1, wherein said associating thecontent with the virtual item and the first user account includesgenerating an entry in a database for storing data regarding the virtualitem and providing a link between the data regarding the virtual itemand a user identifier of the first user account, the link furtherrelated to an entry that includes an identifier of the content.
 3. Themethod of claim 2, wherein the database is accessible to a plurality ofother user accounts to enable an access to the virtual item andretrieval of the content.
 4. The method of claim 2, wherein the databaselinks the first user account and the content to a model of the virtualitem that enables the three-dimensional view of the virtual item in thefirst virtual environment, wherein the first virtual environment is avirtual online store.
 5. The method of claim 4, wherein the model hasgraphics data associated with geometry, shape, shading, and texture ofthe virtual item.
 6. The method of claim 5, wherein the texture isdefined by lighting, a number of polygonal pieces forming the virtualitem, and color.
 7. The method of claim 1, wherein the virtual item isviewable at different perspectives based on gestures made by the firstuser.
 8. The method of claim 1, wherein the virtual item is displayed atdifferent perspectives based on an orientation of the first head-mounteddisplay, wherein the different perspectives are displayed when the firsthead-mounted display is moved by the first user.
 9. The method of claim1, wherein the content is audio content, or video content, or text, orgraphics, or a combination thereof, wherein the content is associatedwith an identity of the first user account accessed to create thecontent.
 10. The method of claim 1, further comprising: receiving, viathe computer network and the second user account, a search request tosearch for the virtual item, wherein the second user account is assignedto the second user; determining whether the second user account haspermission to access the virtual item and the content in response to thesearch request; and sending, via the computer network, the content fordisplay within the shared virtual environment on the second head-mounteddisplay configured to be worn by the second user, wherein said sendingthe data for display of the virtual item and said sending the contentfor display within the shared virtual environment are performed upondetermining that the second user account has permission to access thevirtual item and the content.
 11. The method of claim 1, furthercomprising: sending, via the computer network, the virtual environmentdata for display of the shared virtual environment on the secondhead-mounted display configured to be worn by the second user, whereinthe shared virtual environment includes the tag; receiving, via thecomputer network and the second user account of the second user, anindication of a selection of the tag; determining whether the seconduser account has permission to access the content upon receiving theindication of the selection of the tag; and sending, via the computernetwork and the second user account, content data for displaying thecontent via the second user account upon determining that the seconduser account has the permission.
 12. The method of claim 11, wherein thefirst user account is not accessed via the computer network to displaythe shared virtual environment on the first head-mounted display whenthe second user account is used to access the shared virtual environmenton the second head-mounted display.
 13. The method of claim 1, furthercomprising: sending, via the computer network, the virtual environmentdata for display of the shared virtual environment on the secondhead-mounted display configured to be worn by the second user;receiving, via the computer network and the second user account of thesecond user, an indication of a selection of the virtual item;determining whether the second user account has permission to access thecontent upon receiving the indication of the selection of the virtualitem; sending, via the computer network, content data for displaying thecontent via the second user account upon determining that the seconduser account has the permission.
 14. The method of claim 1, furthercomprising: determining whether the virtual item will be accessed viathe second user account; determining whether the rating exceeds apredetermined rating, wherein said sending the data for display of thevirtual item includes sending; the data for display of the virtual itemin a more prominent manner compared to a manner in which other virtualitems are displayed within the shared virtual environment in response todetermining that the virtual item will be accessed via the second useraccount.
 15. The method of claim 1, further comprising: sending, via thecomputer network, the virtual environment data for display of the sharedvirtual environment on the second head-mounted display, wherein thesecond head-mounted display is configured to be worn by the second user;receiving, via the computer network and the second user account of thesecond user, an indication of a selection of the tag; sending, via thecomputer network, option data for allowing entry of additional contentregarding the virtual item upon receiving the indication of theselection of the tag; receiving, via the computer network, theadditional content; associating the additional content with the virtualitem and the tag.
 16. The method of claim 1, further comprising:sending, via the computer network, the virtual environment data fordisplay of the shared virtual environment on the second head-mounteddisplay, wherein the second head-mounted display is configured to beworn by the second user; receiving, via the computer network and thesecond user account of the second user, an indication of a selection ofthe virtual item; sending, via the computer network, option data forallowing entry of additional content regarding the virtual item uponreceiving the indication of the selection of the virtual item;receiving, via the computer network, the additional content; associatingthe additional content with the virtual item and the tag.
 17. The methodof claim 1, further comprising: counting an amount of content associatedwith the virtual item; determining whether the amount of content exceedsa pre-determined threshold; determining whether access to the virtualenvironment data for display of the shared virtual environment isrequested from the second user account assigned to the second user;sending, via the computer network, the virtual environment data fordisplay of the shared virtual environment on the second head-mounteddisplay configured to be worn by the second user upon determining thatthe access to the virtual environment for display of the shared virtualenvironment is requested from the second user account, wherein theshared virtual environment displays the virtual item in a more prominentlocation compared to a location of display of the virtual item in theshared virtual environment displayed on the first head-mounted display.18. The method of claim 1, wherein the first virtual environmentincludes a virtual user using the virtual item.
 19. The method of claim1, wherein the virtual item is displayed at a first location within theshared virtual environment displayed on the first head-mounted displayand at a second location within the shared virtual environment displayedon the second head-mounted display, wherein the second location isdifferent from the first location.
 20. A system comprising: a firsthead-mounted display, wherein the first head-mounted display isconfigured to be worn by a first user; and one or more servers coupledto the first head-mounted display via a computer network, wherein theone or more servers are configured to send, via the computer network,virtual environment data for display of a shared virtual environment onthe first head-mounted display, wherein the shared virtual environmentincludes a three-dimensional view of a virtual item, wherein the firsthead-mounted display is configured to display the shared virtualenvironment, wherein the one or more servers are configured to receive,via the computer network and a first user account of the first user, anindication of a selection associated with the virtual item displayed onthe first head-mounted display; wherein the one or more servers areconfigured to send, via the computer network, option data for allowingentry of content regarding the virtual item upon receiving theindication of the selection, wherein the first head-mounted display isconfigured to display the option data for the entry of the content,wherein the one or more servers are configured to receive, via thecomputer network, the content; wherein the one or more servers areconfigured to associate the content with the virtual item; and whereinthe one or more servers are configured to send, via the computernetwork, tagged data for displaying a tag associated with the virtualitem, wherein the tag indicates that the content is associated with thevirtual item, wherein the first head-mounted display is configured todisplay the tag, wherein the one or more servers are configured toanalyze a number of interactions with the virtual item including theselection of the virtual item to determine a rating for the virtualitem, wherein the interactions include three-dimensional movement of thevirtual item in the shared virtual environment, wherein the one or moreservers are coupled to a second head-mounted display via the computernetwork, wherein the one or more servers are configured to send, via thecomputer network, data for display of the virtual item within the sharedvirtual environment to the second head-mounted display based on therating and an interest profile associated with a second user account ofa second user.
 21. The system of claim 20, wherein the content is audiocontent, or video content, or text, or a combination thereof, whereinthe content is associated with an identity of the first user accountaccessed to create the content.
 22. The system of claim 20, wherein thesecond head-mounted display is configured to be worn by the second user,wherein the second head-mounted display is configured to display a fieldfor receiving a search request for the virtual item via the second useraccount assigned to the second user, wherein the one or more servers areconfigured to receive, via the computer network and the second useraccount, the search request to search for the virtual item, wherein theone or more servers are configured to determine whether the second useraccount has permission to access the virtual item and the content inresponse to the search request, wherein the one or more servers areconfigured to send, via the computer network, the virtual environmentdata for display of the shared virtual environment upon determining thatthe second user account has permission to access the virtual item andthe content, wherein the second head-mounted display is configured todisplay the shared virtual environment, wherein the shared virtualenvironment includes the content.
 23. The system of claim 20, whereinthe second head-mounted display is configured to be worn by the seconduser, wherein the one or more servers are configured to send, via thecomputer network, the virtual environment data for display of the sharedvirtual environment on the second head-mounted display, wherein theshared virtual environment includes the tag, wherein the one or moreservers are configured to receive, via the computer network and thesecond user account of the second user, an indication of a selection ofthe tag; wherein the one or more servers are configured to determinewhether the second user account has permission to access the contentupon receiving the indication of the selection of the tag, and whereinthe one or more servers are configured to send, via the computernetwork, content data for displaying the content via the second useraccount upon determining that the second user account has thepermission, wherein the second head-mounted display is configured todisplay the content within the shared virtual environment.
 24. Thesystem of claim 23, wherein the first user account is not accessed viathe computer network to display the shared virtual environment on thefirst head-mounted display when the second user account is used toaccess the shared virtual environment on the second head-mounteddisplay.
 25. A system comprising: a first head-mounted displayconfigured to display a shared virtual environment, wherein the firsthead-mounted display is configured to be worn by a first user, whereinthe shared virtual environment includes a three-dimensional view of avirtual item; and a communication device coupled to the firsthead-mounted display for sending, via a computer network and a firstuser account of the first user, an indication of a selection associatedwith the virtual item, wherein the communication device is configured toreceive, via the computer network, option data for allowing entry ofcontent regarding the virtual item after the indication of the selectionis sent, wherein the first head-mounted display is configured to displaythe content, wherein the communication device is configured to send thecontent via the computer network to a server for associating of thecontent with the virtual item, wherein the communication device isconfigured to receive tag data for displaying a tag associated with thevirtual item, wherein the tag indicates that the content is associatedwith the virtual item, a second head-mounted display coupled to thecomputer network, wherein the second head-mounted display is configuredto receive, via the computer network, data for display of the virtualitem within the shared virtual environment based on a rating and aninterest profile associated with a second user account associated with asecond user, wherein the rating is determined from an analysis of anumber of interactions with the virtual item including the selection ofthe virtual item, wherein the interactions include three-dimensionalmovement of the virtual item in the shared virtual environment.
 26. Thesystem of claim 25, wherein the content is audio content, or videocontent, or text, or a combination thereof.